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CHEMICAL  DEMILITARIZATION 

Additional  Management  Actions  Needed  to  Meet  Key 
Performance  Goals  of  DOD's  Chemical 
Demilitarization  Program 


Why  GAO  Did  This  Study 

Destruction  of  the  nation’s 
remaining  stockpile  of  chemical 
weapons  in  a  safe,  efficient,  and 
timely  manner  is  essential  to  meet 
Chemical  Weapons  Convention 
treaty  obligations  and  to  reduce  the 
risk  of  a  potential  catastrophic 
event.  The  Department  of  Defense 
(DOD)  established  the  Chemical 
Demilitarization  Program  to 
manage  the  destruction  of  the 
remaining  stockpile.  GAO  was 
asked  to  evaluate  the  (1)  progress 
DOD  and  the  Army  have  made  in 
addressing  GAO’s  prior 
recommendations  to  strengthen 
program  management, 

(2)  reasonableness  of  schedule 
milestones,  (3)  reliability  of  cost 
estimates,  and  (4)  effectiveness  of 
efforts  to  provide  monetary 
incentives  to  the  systems 
contractors.  GAO  reviewed 
relevant  planning  documents, 
schedules,  cost  estimates,  and 
contracts;  interviewed  program  and 
contractor  officials;  and  visited 
chemical  agent  destruction  sites. 


What  GAO  Recommends 


GAO  recommends  that  DOD  and 
the  Army  develop  interim 
destruction  goals,  approaches,  and 
milestones;  establish  time  frames 
to  complete  its  risk  management 
approach;  develop  realistic 
schedule  and  closure  cost 
estimates;  finalize  and 
independently  review  cost 
estimates;  and  determine  whether  a 
greater  emphasis  can  be  placed  on 
schedule  and  cost,  and  develop 
more  objective  award  fee 
performance  evaluation  criteria. 

DOD  concurred  or  partially 
concurred  with  12  of  GAO’s  13 
recommendations. 

To  view  the  full  product,  including  the  scope 
and  methodology,  click  on  GAO-08-134. 

For  more  information,  contact  Davi  DAgostino 
at  (202)  512-5431  ordagostinod@gao.gov. 


What  GAO  Found 

DOD  and  the  Army  have  taken  steps  in  addressing  GAO’s  prior 
recommendations  to  strengthen  program  management  by  establishing  an 
overall  strategy  and  supporting  implementation  plan,  but  some  key  elements, 
such  as  annual  performance  measures  for  some  key  goals,  including  interim 
destruction  goals,  are  not  fully  developed.  Moreover,  actions  DOD  and  the 
Army  have  taken  to  identify  and  mitigate  the  risk  of  future  program  schedule 
extensions  and  cost  growth  have  not  been  effective  because  the  Chemical 
Materials  Agency’s  risk  management  process  has  not  been  fully  developed  or 
integrated  with  DOD’s  risk  management  process.  As  a  result,  managers  lack 
an  integrated  and  systematic  approach  to  evaluate  and  manage  risk. 

Recently  achieved  destruction  rates  may  indicate  that  adjusted  schedule 
milestones  are  overly  conservative.  Moreover,  the  program’s  schedule  was 
extended  in  2005  in  response  to  the  slower-than-anticipated  destruction  rates 
experienced  since  2003,  when  more  optimistic  milestones  were  adopted. 
GAO’s  review  of  processing  rates  achieved  at  the  sites  since  the  revised 
schedule  determined  that  most  sites  are  significantly  ahead  of  the  revised, 
extended  program  estimates  for  recently  completed  munitions  campaigns.  For 
example,  Umatilla,  Oregon,  completed  destruction  of  one  of  its  rocket 
munitions  by  August  2007,  about  12  months  earlier  than  forecasted. 

GAO  was  not  able  to  verify  the  accuracy  of  the  program’s  cost  estimates 
because  of  shortcomings  in  the  underlying  cost  data,  such  as  undefined 
facility  closure  requirements  and  unstable  baseline  costs.  GAO  determined 
that  the  usefulness  of  the  earned  value  management  tool  to  provide  visibility 
over  the  program’s  cost  performance  is  limited  because  of  these 
shortcomings.  The  program’s  projected  cost  growth  is  largely  attributable  to 
longer  schedules  and  increased  costs  associated  with  facility  closure 
estimates.  GAO  estimated  that  a  1-month  schedule  extension  would  cost  an 
additional  $9.5  million.  Also,  schedules  used  for  the  closure  cost  estimates  for 
destruction  sites  have  increased,  from  12  months  to  about  30  months  at  some 
sites,  but  they  have  not  been  finalized  or  independently  reviewed. 

Since  2003,  the  Army  has  actively  managed  criteria  for  determining  monetary 
fees — largely  award  fees — it  provided  systems  contractors  to  incentivize  their 
performance.  These  fees  had  emphasized  safety  and  environmental 
compliance,  and  Army  officials  acknowledged  that  award  fees  did  not 
successfully  control  schedule  and  cost  growth.  In  2006,  after  audit  results  and 
DOD  direction,  and  while  extending  the  schedule,  the  Army  reinstated 
schedule  as  a  performance  measure.  The  award  fee  plan  allows  site  project 
managers  flexibility  to  weigh  performance  measures  within  a  range,  and  GAO 
determined  that  the  high  end  of  the  range  for  management  was  generally 
selected,  rather  than  for  schedule  and  cost.  Most  criteria  in  the  program’s 
award  fee  plans  are  subjective  and  are  not  linked  to  objective  outcomes  as 
much  as  possible,  as  called  for  in  DOD  guidance.  Given  the  program’s 
maturity,  GAO  found  that  the  Army  could  apply  more  weight  to  performance 

measures,  such  as  schedule  and  cost,  and  add  objective  criteria  in  these  plans. 
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United  States  Government  Accountability  Office 
Washington,  DC  20548 


December  6,  2007 

The  Honorable  Adam  Smith 
Chairman 

The  Honorable  Mac  Thornberry 
Ranking  Member 
Subcommittee  on  Terrorism  and 
Unconventional  Threats  and  Capabilities 
Committee  on  Armed  Services 
House  of  Representatives 

The  Honorable  Jim  Saxton 
House  of  Representatives 

Destruction  of  the  nation’s  remaining  stockpile  of  lethal  chemical  weapons 
in  a  safe,  efficient,  and  timely  manner  is  essential  to  meeting  U.S.  treaty 
obligations  under  the  Chemical  Weapons  Convention  (CWC)1  and  to 
reducing  the  risk  to  the  public  and  the  environment  of  a  potential 
catastrophic  event.  In  1985,  Congress  directed  the  Department  of  Defense 
(DOD)  to  destroy  the  U.S.  stockpile  of  lethal  chemical  agents  and 
munitions  in  the  United  States  with  the  requirement  that  DOD  was  to 
ensure  maximum  protection  for  the  environment,  the  general  public,  and 
the  personnel  involved  in  the  destruction  efforts.  DOD  designated  the 
Department  of  the  Army  as  its  executive  agent,  and  the  Army  established 
the  Chemical  Demilitarization  Program  to  destroy  the  stockpile  at  nine 
storage  sites.2  The  United  States  and  more  than  180  other  countries  are 
signatories  to  the  CWC,  which  went  into  effect  in  1997.  The  CWC  prohibits 


'The  Convention  on  the  Prohibition  of  the  Development,  Production,  Stockpiling  and  Use 
of  Chemical  Weapons  and  on  Their  Destruction  is  commonly  known  as  the  Chemical 
Weapons  Convention. 

JIn  1996,  Congress  directed  DOD  to  evaluate  the  use  of  alternative  technologies  and 
suspended  incineration  planning  activities  at  two  of  the  nine  storage  sites  with  assembled 
weapons — Blue  Grass  Chemical  Agent-Destruction  Pilot  Plant,  Blue  Grass  Chemical 
Activity,  Kentucky,  and  Pueblo  Chemical  Agent-Destruction  Pilot  Plant,  Pueblo  Chemical 
Depot,  Colorado.  See  Pub.  L.  No.  104-201,  §  142  (1996)  and  Pub.  L.  No.  104-208,  §  8065 
(1996).  In  addition,  Congress  directed  that  those  two  sites  be  managed  in  a  program 
independent  from  the  Army’s  Chemical  Demilitarization  Program  and  report  directly  to 
DOD  instead  of  to  the  Army.  See  Pub.  L.  No.  105-261,  §  142  (1998).  The  two  sites  are 
managed  under  a  separate  program  known  as  the  Assembled  Chemical  Weapons 
Alternatives  program,  and  are  not  the  focus  of  this  report. 
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the  use  of  chemical  weapons  and  mandates  a  deadline  of  April  29,  2012, 
for  the  United  States  to  destroy  the  existing  stockpiles.3 

DOD’s  Chemical  Demilitarization  Program  has  not  experienced  a  chemical 
event  since  its  inception  in  1985  that  resulted  in  an  environmental 
catastrophe  or  death,  but  the  program  has  experienced  some  potentially 
serious  events  and  has  been  plagued  by  numerous  schedule  revisions,  cost 
growth,  and  long-standing  management  weaknesses.  The  Chemical 
Demilitarization  Program  is  managed  as  a  major  defense  acquisition 
program  and,  as  such,  has  cost  and  schedule  milestones  that  are  contained 
in  an  Acquisition  Program  Baseline  (APB).  As  of  September  30,  2006, 
Congress  had  appropriated  about  $15.8  billion  for  the  program.  In  2005, 
DOD  reported  that  the  program  had  exceeded  its  milestones  under  the 
2003  APB  in  excess  of  Nunn-McCurdy  thresholds  for  cost  and  schedule 


3The  U.S.  deadline  for  100  percent  destruction  of  the  unitary  stockpile  was  extended  from 
April  29,  2007,  to  April  29,  2012.  The  CWC  allows  an  extension  of  up  to  5  years,  and  an 
extension  was  granted  to  the  United  States  at  its  request  in  December  2006.  For  purposes 
of  this  report,  we  refer  to  the  extended  CWC  deadline  of  April  29,  2012,  as  the  U.S.  treaty 
deadline,  unless  otherwise  indicated. 
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performance4  for  the  second  time  in  3  years,5  and  new  schedule  milestones 
and  cost  estimates  were  being  developed.  These  revisions  increased 
DOD’s  estimated  costs  for  the  entire  Chemical  Demilitarization  Program 
by  33  percent  from  about  $24  billion  in  2003  to  about  $32  billion  in  2005. 
Moreover,  the  Army’s  revised  cost  estimates  for  the  portion  of  the  program 
responsible  for  destroying  the  stockpile,  the  Chemical  Stockpile 
Elimination  Project,6  increased  by  more  than  53  percent  from  about 
$15.8  billion  in  2003  to  about  $24.2  billion  in  2005.  A  large  percentage  of 
this  increase  was  attributable  to  revised  government  cost  estimates  for  the 
destruction  facility  contracts.  In  April  2006,  DOD  announced  that  the 
program  will  not  meet  the  extended  CWC  deadline  of  April  29,  2012,  for 
total  destruction  of  its  stockpile,  and  it  estimated  that  only  66  percent  of 
the  stockpile  will  be  destroyed  by  then.  DOD  also  announced  that  it  would 
seek  resources  to  complete  destruction  as  close  to  the  CWC  deadline  as 
possible  and  separately  requested  authority  to  provide  additional 


4A  Nunn-McCurdy  unit  cost  breach  occurs  when  a  major  defense  acquisition  program 
experiences  an  increase  of  at  least  15  percent  in  program  acquisition  unit  cost  or 
procurement  unit  cost  above  the  unit  costs  in  the  current  APB.  Unit  cost  reporting  is 
required  by  10  U.S.C.  §  2433.  For  programs  with  unit  cost  increases  of  at  least  25  percent,  a 
certification  by  the  Secretary  of  Defense  is  required.  Certification  responsibility  has  been 
delegated  to  the  Under  Secretaiy  of  Defense  for  Acquisition,  Technology,  and  Logistics, 
who,  by  law,  must  certify  that  (1)  such  an  acquisition  program  is  essential  to  national 
security;  (2)  there  are  no  alternatives  to  such  an  acquisition  program,  which  will  provide 
equal  or  greater  military  capability  at  less  cost;  (3)  the  new  estimates  of  the  program 
acquisition  unit  cost  or  procurement  unit  cost  are  reasonable;  and  (4)  the  management 
structure  for  the  acquisition  program  is  adequate  to  manage  and  control  program 
acquisition  unit  cost  or  procurement  unit  cost.  See  10  U.S.C.  §  2433  (b)(2)(A)  (2007). 

5Total  program  cost  estimates  exceeded  APBs  in  2001  and  again  in  2005.  On  May  2,  2002, 
the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics — the  Defense 
Acquisition  Executive — submitted  a  Nunn-McCurdy  breach  certification  to  Congress  for 
the  Chemical  Demilitarization  Program  as  required  by  10  U.S.C.  §  2433.  The  unit  cost 
breach  was  due  primarily  to  (1)  revised  processing  rates;  (2)  schedule  extensions;  (3)  new 
environmental  regulations;  (4)  worse-than-expected  condition  of  the  stockpile;  (5)  increase 
in  equipment,  labor  rates,  and  construction  costs;  and  (6)  higher  emergency  preparedness 
costs.  On  May  6,  2005,  the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and 
Logistics  again  certified  to  Congress  that  the  program  acquisition  unit  costs  for  the 
Chemical  Demilitarization  Program  had  increased  41  percent  (in  then-year  dollars)  due 
primarily  to  more  realistic  schedule  estimates  associated  with  operations  and  closure 
phases,  environmental  regulatory  compliance,  and  concept/design  maturation.  Program 
acquisition  unit  costs  for  the  Newport  Chemical  Agent  Disposal  Facility,  Newport  Chemical 
Depot,  Indiana,  program  increased  by  52  percent  (in  then-year  dollars)  due  primarily  to 
more  realistic  schedule  estimates  associated  with  the  nerve  agent  neutralization  process 
and  secondary  hazardous  waste  generation. 

6The  Chemical  Stockpile  Elimination  Project  includes  the  program’s  four  operating 
incineration  sites  and  two  bulk-agent-only  neutralization  sites. 
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monetary  incentives  if  systems  contractors  operating  the  incineration  sites 
could  accelerate  their  destruction  activities  to  safely  exceed,  meet,  or 
finish  destruction  near  the  CWC  deadline.  Moreover,  the  program’s 
remaining  four  incineration  sites — Anniston  Chemical  Agent  Disposal 
Facility,  Anniston  Chemical  Activity  (Anniston),  Alabama;  Pine  Bluff 
Chemical  Agent  Disposal  Facility,  Pine  Bluff  Chemical  Activity  (Pine 
Bluff),  Arkansas;  Tooele  Chemical  Agent  Disposal  Facility,  Deseret 
Chemical  Depot  (Tooele),  Utah;  and  Umatilla  Chemical  Agent  Disposal 
Facility,  Umatilla  Chemical  Depot  (Umatilla),  Oregon — were  forecasted  to 
miss  the  CWC  deadline  for  the  first  time.7 

In  the  past,  the  program  has  had  long-term  problems  in  meeting  all  of  the 
schedule  milestones  for  destroying  the  stockpile,  in  part  because  of  the 
lengthy  environmental  permitting  processes  prior  to  operation  start-up 
and  unexpected  conditions  of  the  aging  munitions.  The  stockpile 
destruction  program  managed  by  the  Chemical  Materials  Agency  (CMA) 
has  now  reached  a  more  mature  phase,  and  CMA  and  its  systems 
contractors  have  more  experience  in  incinerating  each  type  of  chemical 
agent  and  munition  in  its  stockpile  as  well  as  neutralizing  bulk  nerve  and 
blister  agents. 

DOD,  the  Army,  CMA,  and  the  systems  contractors  that  operate  the 
destruction  facilities  face  a  difficult  and  hazardous  task,  as  well  as  the 
need  to  overcome  some  significant  technical  challenges,  in  destroying  the 
U.S.  stockpile  of  lethal  chemical  agents  while  ensuring  the  safety  of 
workers,  the  public,  and  the  environment.  Recognizing  these  difficulties 
and  the  unique  program  risks,  DOD  uses  a  contract  that  reimburses  the 
systems  contractors  for  all  reasonable,  allowable,  and  allocable  costs.  To 
encourage  defense  contractors  to  perform  in  an  innovative,  efficient,  and 
effective  way  in  areas  deemed  important  to  the  program’s  success,  DOD 
gives  its  systems  contractors  the  opportunity  to  earn  millions  of  dollars 
through  monetary  incentives  known  as  award  and  incentive  fees. 

We  reported  in  2003  that  DOD  and  the  Army  had  not  developed  an  overall 
strategy  and  risk  management  approach  for  the  chemical  demilitarization 
program  to  guide  the  program  with  a  set  of  principles  and  a  means  to 


'These  four  incineration  sites;  the  neutralization  site  at  Newport  Chemical  Agent  Disposal 
Facility,  Newport  Chemical  Depot,  Indiana;  and  the  two  closed  sites  that  were  at  Johnston 
Atoll  in  the  Pacific  Ocean  and  Aberdeen  Chemical  Agent  Disposal  Facility,  Maryland,  are 
managed  for  the  Army  by  the  Chemical  Materials  Agency,  which  was  formerly  known  as 
the  Program  Manager  for  Chemical  Demilitarization. 
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manage  risk,  understand  its  evolution  and  implementation,  and  determine 
whether  initiatives  are  achieving  their  desired  results.8  The  absence  of  an 
overall  strategy  and  risk  management  approach  left  the  program  without  a 
clear  road  map  to  monitor  program  performance  and  meet  the  goal  of 
destroying  the  chemical  stockpile  in  a  safe  and  timely  manner.  We 
recommended  that  DOD  and  the  Army  adopt  practices  that  leading 
organizations  embrace  for  effectively  implementing  and  managing 
programs,  to  include  promulgating  a  comprehensive  mission  statement, 
long-term  and  annual  performance  goals,  measurable  performance 
indicators,  and  evaluation  and  corrective  action  plans.  DOD  concurred 
with  our  recommendations  and  stated  that  it  would  establish  these 
management  tools.  A  list  of  related  GAO  products  regarding  chemical 
demilitarization  and  storage  and  disposal  of  the  chemical  weapons 
stockpile  is  included  at  the  end  of  this  report. 

We  also  reported  that  overall  DOD  programs  have  paid  contractors  large 
amounts  of  award  fees  on  acquisitions  that  are  falling  behind  schedule, 
overrunning  costs,  and  experiencing  significant  technical  problems.9  In 
addition,  we  recommended,  among  other  things,  that  DOD  instruct  the 
military  services  to  move  toward  more  outcome-based  award  fee  criteria 
that  are  both  achievable  and  promote  accountability  for  acquisition 
outcomes.10  DOD  concurred  with  the  recommendation  and,  in  a  March 
2006  policy  memorandum,  addressed  desired  outcomes  and  the  role  the 
award  fee  should  play  in  the  overall  acquisition  strategy.  The 
memorandum  stated  that  “while  award  fee  contracts  are  used  when  it  is 
neither  feasible  nor  effective  to  devise  predetermined  objective 
performance  targets,  it  is  imperative  that  award  fees  be  tied  to  identifiable 
interim  outcomes,  discrete  events  or  milestones,  as  much  as  possible.” 

This  report  responds  to  a  request  from  the  House  Committee  on  Armed 
Services’  Subcommittee  on  Terrorism  and  Unconventional  Threats  and 
Capabilities  that  we  conduct  a  review  of  DOD’s  Chemical  Demilitarization 
Program.  Specifically,  this  report  evaluates  the  (1)  progress  DOD  and  the 


sGAO,  Chemical  Weapons:  Sustained  Leadership,  Along  with  Key  Strategic  Management 
Tools,  Is  Needed  to  Guide  DOD’s  Destruction  Program,  GAO-03-1031  (Washington,  D.C.: 
Sept.  5,  2003). 

°GAO,  Defense  Acquisitions:  DOD  Wastes  Billions  of  Dollars  through  Poorly  Structured 
Incentives,  GAO-06-409T  (Washington,  D.C.:  Apr.  5,  2006). 

10GAO,  Defense  Acquisitions:  DOD  Has  Paid  Billions  in  Award  and  Inceiitive  Fees 
Regardless  of  Acquisition  Outcomes,  GAO-06-66  (Washington,  D.C.:  Dec.  19,  2005). 
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Army  have  made  in  addressing  our  prior  recommendations  to  strengthen 
program  management,  whether  this  progress  has  been  sufficient  to  correct 
past  program  weaknesses,  and  the  effectiveness  of  actions  DOD  and  the 
Army  have  taken  to  identify  and  mitigate  the  risk  of  future  program 
schedule  extensions  and  cost  growth;  (2)  reasonableness  of  the 
methodology  used  to  determine  schedule  milestones;  (3)  reliability  of  its 
cost  estimates;  and  (4)  effectiveness  of  DOD’s  and  the  Army’s  use  of 
monetary  incentives  to  improve  systems  contractors’  performance. 

To  assess  the  progress  DOD  and  the  Army  have  made  in  implementing 
GAO’s  prior  recommendations  to  strengthen  program  management, 
whether  it  has  been  sufficient,  and  the  effectiveness  of  actions  taken  to 
identify  and  mitigate  the  risk  of  future  program  schedule  extensions  and 
cost  growth,  we  obtained  and  reviewed  strategic  and  implementation 
plans  and  interviewed  cognizant  officials  to  identify  whether  key  elements 
were  present,  such  as  a  program  mission  statement,  long-term  goals  and 
objectives,  delineation  of  roles  and  responsibilities  of  DOD  and  Army 
offices,  and  near-term  performance  measures.  To  assess  the 
reasonableness  of  the  methodology  used  to  determine  schedule  milestones 
as  well  as  the  reliability  of  cost  estimates,  we  reviewed  current  program 
estimates,  destruction  schedules,  earned  value  management  (EVM)11  data, 
and  other  documents.  We  also  obtained  and  reviewed  the  program’s  risk 
management  plans  and  related  documents.  In  addition,  we  identified  the 
issues  that  had  caused  delays  and  ascertained  approaches  being  used  to 
reduce  the  potential  for  delays  in  the  future.  To  analyze  the  effectiveness 
of  DOD’s  and  the  Army’s  use  of  monetary  incentives  to  improve  systems 
contractors’  performance,  we  obtained  and  reviewed  contract  documents 
on  the  award  fees  offered  to  the  systems  contractors,  and  interviewed 
DOD  and  Army  officials  as  well  as  systems  contractors’  representatives. 

We  obtained  and  reviewed  the  incentive  agreement  plan  and  incentive  fees 
for  the  Aberdeen  Chemical  Agent  Disposal  Facility  (Aberdeen),  Maryland, 
and  Newport  Chemical  Agent  Disposal  Facility,  Newport  Chemical  Depot 
(Newport),  Indiana,  to  determine  the  challenges  those  sites  faced  in 
negotiating  the  incentive  fee  agreements.  Also,  we  interviewed  officials 


uThe  Office  of  Management  and  Budget  (OMB)  defines  EVM  as  a  project  management  tool 
that  effectively  integrates  the  project  scope  of  work  with  cost,  schedule,  and  technical 
performance  elements  for  optimal  project  planning  and  control.  Beginning  in  August  2005, 
OMB  specifically  required  federal  agencies  and  contractors  to  use  EVM  systems  on  all  new 
capital  assets.  See  Office  of  Management  and  Budget,  Capital  Programming  Guide 
(Washington,  D.C.:  2006).  Also,  DOD  requires  the  use  of  EVM  on  all  programs  greater  than 
$20  million. 
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with  the  U.S.  Army  Sustainment  Command,  CMA,  and  CMA  and  systems 
contractors  at  Tooele  and  Umatilla.  In  addition,  we  met  with  DOD  and 
Army  program  officials  and  interviewed  officials  at  two  of  the  chemical 
agent  destruction  sites. 

This  report  focuses  primarily  on  the  seven  CMA-managed  chemical 
destruction  sites — Aberdeen,  Anniston,  Johnston  Atoll  in  the  Pacific 
Ocean,  Newport,  Pine  Bluff,  Tooele,  and  Umatilla.  We  selected  these  sites 
for  our  review  because  most  are  in  the  operational  phase,  represent  about 
82  percent  of  the  remaining  stockpile  of  chemical  agents  to  be  destroyed, 
and  have  reached  a  level  of  maturity  in  processing  a  variety  of  types  of 
munitions  and  agents  that  will  help  meet  CWC  goals.  Two  sites — Aberdeen 
and  Johnston  Atoll — have  already  completed  operations  and  have  either 
been  closed  or  are  being  closed.  This  report  does  not  focus  on  the  two 
chemical  agent  destruction  sites  that  are  part  of  the  Assembled  Chemical 
Weapons  Alternatives  (ACWA)  program — Blue  Grass  Chemical  Agent- 
Destruction  Pilot  Plant,  Blue  Grass  Chemical  Activity  (Blue  Grass), 
Kentucky,  and  Pueblo  Chemical  Agent-Destruction  Pilot  Plant,  Pueblo 
Chemical  Depot  (Pueblo),  Colorado — because  they  are  currently  in  the 
design  and  construction  phase  to  destroy  chemical  agents,  report  directly 
to  DOD  rather  than  to  the  Army,  and  because  these  facilities  are  not 
forecasted  to  begin  destruction  operations  before  the  CWC  deadline, 
although  the  stockpiles  at  these  two  sites  are  required  to  be  destroyed  by 
the  deadline  as  well. 

We  examined  the  reliability  of  the  data  used  in  this  report  by  verifying 
EVM  system  certification,  applying  analytical  tests  to  the  EVM  data,  and 
examining  EVM  system  effectiveness  in  providing  meaningful  performance 
measurement  to  program  management.  We  determined  that  the  EVM 
system  as  used  at  each  site  was  not  reliable,  which  is  discussed  more  fully 
in  the  report.  We  conducted  this  performance  audit  from  May  2006 
through  July  2007  in  accordance  with  generally  accepted  government 
auditing  standards.  A  more  thorough  description  of  our  scope  and 
methodology  is  provided  in  appendix  I. 


Results  in  Brief 


While  DOD  and  the  Army  have  taken  steps  toward  addressing  our  prior 
recommendations  to  strengthen  program  management  by  establishing 
strategic  and  supporting  implementation  plans,  they  have  not  fully 
developed  an  integrated  approach  for  measuring  performance  or  for 
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managing  risk.12  DOD  issued  a  strategic  plan  in  June  2005,  and  CMA  issued 
a  strategic  plan  in  July  2005  and  a  risk  management  plan  in  May  2006. 

While  these  plans  identify  challenges  and  contain  performance  measures 
to  monitor  progress,  we  found  that  the  plans  lack  schedule  goals  for  key 
phases.  Neither  DOD’s  2006  baseline  nor  its  April  2006  and  April  2007 
Selected  Acquisition  Reports  (SAR)  to  Congress  include  schedule 
milestones  extending  beyond  2012.  The  program  further  lacks  annual 
performance  goals  linked  to  long-term  strategic  goals,  such  as  meeting  the 
extended  April  29,  2012,  CWC  deadline.  Without  this  linkage,  program 
officials  will  lack  the  information  on  interim  progress  needed  to  determine 
whether  corrective  action  is  needed  to  safely  destroy  the  stockpile  by  the 
treaty  deadline.  Furthermore,  while  CMA’s  integrated  risk  management 
plan  identified  a  process  for  conducting  risk  management,  the  agency  has 
not  developed  the  tools  needed  to  implement  its  approach,  such  as  a  risk 
trade-off  analysis.  For  example,  the  program’s  plans  identified 
contamination  of  the  mustard  agent  stockpile  at  one  site  as  a  key  risk  but 
lacked  a  method  for  analyzing  that  risk  or  managing  its  potential  effects  on 
cost  and  schedule.  The  later  discovery  of  the  extent  of  the  contamination 
delayed  planned  operations  by  more  than  12  months.  Also,  our  analysis 
showed  that  DOD’s  and  the  Army’s  processes  for  identifying  risk  are  not 
integrated  across  sites  or  organizations,  and  CMA’s  identified  risks  have 
not  been  fully  aligned  with  those  identified  in  DOD’s  strategic  plan. 
Consequently,  program  managers  are  hampered  in  their  information 
sharing  and  their  progress  in  safely  destroying  chemical  agents,  as  well  as 
in  their  ability  to  systematically  evaluate  and  manage  risk.  Finally,  unlike 
more  useful  annual  performance  plans  we  have  reviewed,  which  include 
historical  performance  trends  and  multiyear  future  goals,  CMA’s  annual 
performance  plan  has  a  single-year  focus.  We  are  recommending  that  the 
Secretary  of  the  Army  incorporate  historical  trends  and  multiyear  future 
goals  in  its  annual  performance  plan;  develop  interim  destruction  goals, 
approaches,  and  milestones;  and  establish  a  time  frame  for  implementing 
its  risk  management  approach,  including  integration  across  sites  and  with 
DOD. 

Recently  achieved  processing  rates  have  considerably  exceeded 
forecasted  rates  at  the  operational  sites  and  indicate  that  DOD’s  revised 
schedule  milestones  may  be  too  conservative  to  be  a  realistic  measure. 
DOD  extended  its  Chemical  Demilitarization  Program  schedule  in  2005  in 
response  to  the  slower  actual  destruction  rates  experienced  than  had  been 


12GAO’s  prior  recommendations  are  found  in  GAO-03-1031,  p.  26. 
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estimated  when  more  optimistic  milestones  were  adopted  in  2003.  As  we 
have  previously  reported,13  best  practices  for  schedule  development  call 
for  assumptions  that  are  both  up-to-date  and  reasonable.  Using  a  new 
schedule  methodology,  and  as  a  result  of  factoring  in  slower  processing 
rates  and  greater  risk,  CMA  extended  schedules  at  each  site  between  62 
months  and  80  months  beyond  the  baselines  established  in  2003.  The 
extensions  prompted  budget  and  acquisition  reviews,  including  an 
independent  cost  and  schedule  assessment,  which  in  turn  led  DOD  to 
further  extend  the  program’s  schedule  milestones  for  completing 
destruction  operations  and  closure  activities.  Our  analysis  of  recently 
achieved  processing  rates  showed  that  at  the  time  of  our  review,  the  sites 
were  between  2  months  and  39  months  ahead  of  the  revised  estimates  for 
munitions  destruction.  Revised  schedule  estimates  published  in  the  April 
2007  SAR  further  reflect  processing  rates  that  considerably  exceed 
forecasts.  Nevertheless,  DOD  has  no  current  plans  to  reassess  the  program 
schedule  milestones,  as  such  reassessments  are  usually  prompted  by 
schedule  slippages.  Higher  destruction  rates  than  scheduled  indicate  that 
an  independent  reassessment  of  the  program’s  scheduled  milestones  may 
be  warranted,  thus  assuring  DOD  and  Congress  of  the  reasonableness  of 
the  baseline  against  which  its  progress  is  being  measured  and  of  its  costs 
as  the  current  schedule  is  not  providing  DOD  and  Congress  with  a 
reasonable  baseline  against  which  to  measure  progress  and  cost.  We  are 
recommending  that  the  Secretary  of  Defense  direct  the  Cost  Analysis 
Improvement  Group  to  assess  the  reasonableness  of  schedule  milestones, 
and  that  DOD  periodically  review  the  program’s  processing  rates  to 
determine  whether  the  milestones  are  still  reasonable  and  include 
schedule  milestones  for  completing  destruction  operations  and  facility 
closure  activities  in  the  program’s  baseline  and  acquisition  report. 

Shortcomings  in  the  underlying  data  used  in  the  program’s  EVM  showed 
that  the  cost  estimate  is  unreliable.  Our  review  showed  that  performance 
baselines  lacked  validity  and  had  not  been  stable  because  all  requirements, 
such  as  closure  costs,  are  not  fully  identified  and  included  in  program  cost 
estimates  and  because  large  amounts  of  additional  costs  are  added  to  the 
performance  baseline  annually.  In  addition,  systems  contractors’  cost 
estimates  were  significantly  lower  than  the  Army’s  cost  estimates,  due 
largely  to  the  Army’s  expectations  of  longer  operational  periods  and 
greater  closure  requirements.  The  program’s  cost  increases  are  largely 


13GAO,  Cost  Assessment  Guide:  Best  Practices  for  Estim  ating  and  Managing  Program 
Costs, GAO-07-1 134SP  (Washington,  D.C.:  July  2007,  Exposure  Draft). 
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attributable  to  extended  schedules  and  facility  closure  estimates.  Best 
practices  indicate  that  cost  estimates  should  be  comprehensive,  and  DOD 
guidance  further  reflects  the  need  for  valid  and  stable  performance 
baselines  from  which  to  make  projections  and  perform  trend  analysis. 

DOD  and  Congress  currently  lack  accurate  estimates  for  budgeting  for  and 
funding  all  activities  through  facility  closure.  Similarly,  given  the 
significant  differences  between  the  contractors’  and  Army’s  cost 
estimates,  an  independent  cost  review  would  provide  DOD  and  Congress 
assurance  that  cost  estimates  are  realistic.  We  are  recommending  that  the 
Army  define  closure  requirements  and  develop  accurate  and  realistic 
closure  cost  estimates;  define  all  contract  requirements,  including 
authorized  unpriced  work  associated  with  facility  closure  activities,  and 
establish  a  time  frame  for  placing  these  requirements  on  contract;  and  that 
an  independent  review  of  the  cost  estimates  be  done  and  linked  with 
reasonable  schedule  milestones. 

According  to  the  Army  and  based  on  our  analysis,  award  fees — the 
predominant  fees — offered  to  contractors  have  thus  far  not  satisfactorily 
resulted  in  the  control  of  schedule  performance  outcomes  and  cost 
growth.  Army  officials  indicate  that  the  majority  of  the  award  fee 
incentives  are  dedicated  to  achieving  high  standards  for  safety,  surety,  and 
environmental  compliance  in  recognition  of  the  lethal  nature  of  the 
chemical  agent  stockpile  and  the  congressional  mandate  for  maximum 
protection  of  the  workforce,  the  public,  and  the  environment.  The  Army 
has  actively  reviewed  and  adjusted  criteria  for  determining  award  fees  it 
has  provided  to  the  systems  contractors  to  incentivize  aspects  of  their 
performance.  Based  on  our  analysis,  from  2003  through  2006,  the  award 
fee  plans  have  heavily  emphasized  safety  and  environmental  compliance 
and  did  not  have  a  separate  performance  measure  for  schedule  until  2006. 
Army  officials  acknowledged  that  award  fees  have  not  satisfactorily 
resulted  in  the  control  of  schedule  and  cost  growth.  In  response  to  our 
previous  recommendations,  in  March  2006,  DOD  issued  policy  guidance 
directing  that  award  fees  be  tied  to  identifiable  interim  outcomes,  discrete 
events,  or  milestones  as  much  as  possible.  CMA  issued  its  award  fee  policy 
in  September  2006,  and  our  review  of  this  policy  indicated  that  CMA  did 
not  clearly  or  fully  incorporate  the  concept  of  tying  “award  fees  to 
identifiable  interim  outcomes,  discrete  events  or  milestones  as  much  as 
possible,”  as  directed  in  the  March  2006  DOD  guidance.  In  addition,  the 
current  award  fee  plan  for  the  five  chemical  demilitarization  sites  we 
reviewed  allows  government  site  project  managers  flexibility  in  weighting 
each  performance  measure  within  a  set  range.  We  found  that  managers 
have  selected  the  high  end  of  the  range  for  management,  and  the  low  end 
of  the  range  for  schedule  and  cost,  when  determining  award  fee 
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percentages.  Moreover,  many  performance  measures  in  the  award  fee 
plans  contain  subjective  award  fee  criteria,  and  award  fees  for  the  most 
part  are  not  linked  to  objective  or  measurable  criteria.  Our  analysis 
showed  that  it  is  possible  for  CMA  to  further  link  award  fee  criteria  in  this 
program  to  objective  or  measurable  outcomes  in  at  least  three 
performance  measures.  Given  the  maturity  of  the  operations  phase  at  the 
incineration  sites  and  Newport,  although  not  legally  required,  we  identified 
opportunities,  from  a  management  perspective,  to  apply  more  weight  to 
performance  measures,  such  as  schedule  and  cost,  to  better  align  CMA’s 
award  fee  policy  with  DOD’s  policy,  and  to  make  greater  use  of  objective, 
measurable  criteria  in  award  fee  plans  to  increase  the  likelihood  of 
meeting  the  April  29,  2012,  treaty  deadline.  Two  recently  established  and 
planned  performance-based  incentive  fees  for  the  incineration  sites  are 
focused  on  schedule.  One  fee  was  established  in  2005  to  encourage  and 
reward  collaboration  among  the  incineration  site  systems  contractors  with 
the  goal  of  improving  the  overall  program  schedule  and  cost  performance. 
This  fee  is  awarded  based  on  sites’  collective  performance  and,  in  fiscal 
year  2007,  this  incentive  fee  focused  mostly  on  schedule  performance 
associated  with  achieving  the  CWC  treaty  deadline.  The  other  planned 
incentive  fee,  as  authorized  by  Congress,  is  structured  to  accelerate 
destruction  operations  and  facility  closure  at  the  incineration  sites.  We 
could  not  determine  how  effective  these  incentives  might  be  due  to  the 
collective  manner  in  which  the  fee  to  encourage  and  reward  collaboration 
is  earned,  its  relative  size,  and  recent  focus  on  objective  schedule 
performance,  and  because  the  other  planned  incentive  fee  has  not  yet 
been  implemented.  We  are  recommending  that  the  Army  (1)  determine 
whether  greater  weight  can  be  applied  to  key  performance  measures,  such 
as  schedule  and  cost,  in  award  fee  plans;  (2)  better  align  its  award  fee 
policy  with  DOD’s  March  2006  guidance  on  award  fees;  and  (3)  link  award 
fee  criteria,  as  much  as  possible,  to  performance  measures  that  focus  on 
identifiable  interim  outcomes,  discrete  events,  or  milestones. 

We  provided  a  draft  of  this  report  to  DOD  in  October  2007  for  its  review 
and  comment.  In  written  comments  on  a  draft  of  this  report,  DOD 
concurred  or  partially  concurred  with  12  of  our  13  recommendations.  DOD 
did  not  concur  with  our  recommendation  regarding  defining  closure 
requirements  and  developing  accurate  and  realistic  closure  cost  estimates 
for  each  of  DOD’s  chemical  demilitarization  sites.  DOD  recommended  that 
we  delete  it,  since  DOD  officials  believe  that  a  subsequent 
recommendation  defining  all  contract  requirements,  including  those 
associated  with  facility  closure  activities,  was  more  encompassing.  Based 
on  DOD’s  comments,  rather  than  deleting  the  recommendation,  we 
clarified  it  and  the  related  recommendation  to  more  clearly  distinguish  the 
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difference  between  the  two  recommendations,  and  consolidated  it  with 
our  two  other  recommendations  related  to  defining  contract  requirements. 
DOD  also  stated  that  the  department  will  continue  to  use  acquisition 
management  tools  and  discipline  to  ensure  the  destruction  of  the  United 
States’  chemical  weapons  stockpile  in  a  safe  and  secure  manner,  while 
being  economical  and  meeting  the  U.S.  commitments  under  the  CWC. 

DOD  identified  a  number  of  actions  that  it  already  has  initiated  in  response 
to  our  recommendations.  DOD  also  provided  us  with  technical  comments, 
which  we  incorporated  in  the  report  as  appropriate.  DOD’s  response  is 
reprinted  in  appendix  III. 


Background 


In  1985,  Congress  directed  DOD  to  destroy  the  U.S.  stockpile  of  lethal 
chemical  agents  and  munitions  in  the  United  States.14  DOD  designated  the 
Department  of  the  Army  as  its  executive  agent  for  the  program,  and  the 
Army  established  the  Chemical  Demilitarization  Program,  which  was 
charged  with  the  destruction  of  the  original  stockpile  at  nine  storage  sites. 
Figure  1  shows  the  locations  of  the  nine  storage  sites;  the  number  of  items, 
such  as  projectiles,  rockets,  mortars,  land  mines,  and  bombs,  originally 
stored  at  each  site;  and  the  number  of  items  remaining  to  be  destroyed  as 
of  September  30,  2007. 


14The  Department  of  Defense  Authorization  Act  for  Fiscal  Year  1986,  Pub.  L.  No.  99-145,  § 
1412(a)  (1985). 
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Figure  1:  U.S.  Stockpile  of  Chemical  Agents  and  Munitions 


The  remaining  U.S.  stockpile,  as  of  June  2007,  consists  of  two  nerve 
agents — GB  and  VX15 — and  a  blister  agent — mustard16 — which  currently 


15Nerve  agents  are  the  most  toxic  and  rapidly  acting  of  known  chemical  warfare  agents. 
GB,  also  known  as  sarin,  and  VX  Eire  human-made  lethal  chemical  warfare  nerve  agents 
that  affect  the  nervous  system  by  interfering  with  the  signals  sent  from  the  brain  to  vital 
orgEins. 

16Blister  agents  include  mustard,  which  is  a  type  of  chemical  warfare  agent  and  powerful 
irritant  that  causes  blistering  of  the  skin  and  mucous  membranes  on  contact. 
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are  stored  at  seven  sites  in  the  continental  United  States.  Incineration  was 
initially  selected  as  the  method  to  destroy  the  stockpile  at  all  nine  sites, 
and  destruction  began  in  1990  at  Johnston  Atoll.  This  location  was 
selected  to  be  the  operational  prototype  facility  to  test  the  destruction 
process  prior  to  it  being  conducted  in  the  continental  United  States.  Figure 
2  shows  an  example  of  a  chemical  agent  disposal  facility  like  those  used  at 
the  four  operational  incineration  sites  (Anniston,  Pine  Bluff,  Tooele,  and 
Umatilla). 


Figure  2:  Umatilla  Chemical  Agent  Disposal  Facility,  Umatilla  Chemical  Depot,  Oregon 


Source:  Umatilla  Chemical  Agent  Disposal  Facilitiy. 


Subsequent  to  the  decision  to  use  incineration  to  destroy  the  remaining 
stockpile,  Congress  directed  DOD  to  evaluate  the  possibility  of 
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using  alternative  technologies  to  incineration.17  In  1994,  the  Army  initiated 
a  project  to  develop  nonincineration  technologies  for  use  at  the 
two  bulk-agent-only  sites  at  Aberdeen  and  Newport.  These  sites  were 
selected  in  part  because  their  stockpiles  were  relatively  simple — each  site 
had  only  one  type  of  agent  and  this  agent  was  stored  in  bulk-agent  (ton) 
containers,  which  are  used  to  store  various  chemical  agents  at  the 
chemical  agent  destruction  sites.18  Figure  3  shows  a  ton  container  being 
transported  to  the  processing  building  at  Newport. 


17Pub.  L.  No.  104-201,  §  142  (1996). 

18Since  the  1930s,  the  Army  and  the  chemical  industry  have  used  ton  containers  to  store 
and  ship  bulk  chemicals,  including  chemical  agent.  Equivalent  in  length  and  diameter  to 
two  stacked  55-gallon  drums,  a  ton  container  weighs  approximately  1,600  pounds  and 
measures  nearly  7  feet  in  length. 
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Figure  3:  Delivery  of  a  Ton  Container  to  the  Processing  Building  at  the  Newport  Chemical  Agent  Disposal  Facility,  Newport 
Chemical  Depot,  Indiana 


Source:  Newport  Chemical  Agent  Disposal  Facility. 
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In  1997,  DOD  approved  pilot  testing  of  a  neutralization  technology  at  these 
two  sites.  Figure  4  shows  the  type  of  hydrolysate-filled  intermodal  storage 
containers  used  at  Newport.  The  hydrolysate  is  awaiting  final  disposition.19 


19The  stockpile  at  the  Newport  Chemical  Depot,  Indiana,  is  the  first  U.S.  stockpile 
containing  VX  that  will  be  destroyed  by  using  neutralization — a  process  that  mixes  hot 
water  and  sodium  hydroxide  (a  caustic  chemical)  with  VX  to  change  the  chemical 
composition  to  a  less  toxic  form.  The  resulting  by-product  is  a  liquid  wastewater  commonly 
referred  to  as  hydrolysate  that  consists  mostly  of  water  but  also  has  a  caustic  component 
and  organic  salts  that  need  further  treatment  to  meet  CWC  requirements  and  to  meet 
federal  and  state  environmental  requirements  for  disposal.  The  hydrolysate  is  being  stored 
in  intermodal  storage  containers  on-site  until  a  post-treatment  plan  can  be  implemented. 
For  additional  information  on  the  disposal  of  hydrolysate,  see  GAO,  Chemical 
Demilitarization:  Actions  Needed  to  Improve  the  Reliability  of  the  Army’s  Cost 
Comparison  Analysis  for  Treatment  and  Disposal  Options  for  Newport  VX  Hydrolysate, 
GAO-07-240R  (Washington,  D.C.:  Jan.  26,  2007). 
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Figure  4:  Intermodal  Storage  Containers  at  the  Newport  Chemical  Agent  Disposal  Facility,  Newport  Chemical  Depot,  Indiana 


Source:  Newport  Chemical  Agent  Disposal  Facility. 


Also  in  1996,  Congress  directed  DOD  to  evaluate  the  use  of  alternative 
technologies  and  suspended  incineration  planning  activities  at  Blue  Grass 
and  Pueblo,  two  sites  with  assembled  weapons.20  Furthermore,  Congress 
directed  that  these  two  sites  be  managed  in  a  program  independent  of  the 
Army’s  Chemical  Demilitarization  Program  and  report  to  DOD  directly 
instead  of  the  Army.21  Thus,  the  ACWA  program  was  established. 

Table  1  shows  DOD’s  nine  chemical  agent  stockpile  sites;  the  primary 
destruction  method  used  (i.e.,  incineration  or  neutralization)  and  the 


20Pub.  L.  No.  104-201,  §  142  (1996). 
21Pub.  L.  No.  105-261,  §  142  (1998). 
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secondary  disposal  method  used  for  the  treatment  of  the  resulting 
wastewater,  if  applicable;  the  types  of  munitions  destroyed,  such  as 
projectiles,  rockets,  mortars,  land  mines,  and  bombs  in  addition  to  ton 
containers;  the  types  of  agents  (GB,  VX,  and  mustard)  and  their  status 
(e.g.,  destroyed,  ongoing  destruction,  or  planned  destruction);  the  original 
agent  tonnage;  the  tonnage  destroyed;  the  percentage  of  the  stockpile 
destroyed;  the  remaining  agent  tonnage;  and  the  status  of  each  facility 
(closed,  operational,  or  under  construction)  as  of  September  30,  2007. 


Table  1:  Status  of  DOD’s  Chemical  Agent  Stockpile  Sites  and  Chemical  Agent  Destroyed  as  of  September  30,  2007 


Site 

Destruction 

method 

Munition  types 
destroyed 

Type  of  agent 
and  status9 

Original 

agent 

tonnage 

Tonnage 

destroyed 

Percentage 
of  stockpile 
destroyed 

Remaining 
agent 
tonnage 
(as  of 
September 
2007) 

Facility 

status 

Johnston 

Atoll 

Incineration 

Projectiles,  land 
mines,  mortars, 
and  ton 
containers 

GB,  VX,  and 

mustard 

(destroyed) 

705b 

705 

100 

0 

Closed 

Tooele 

Incineration 

Ton  containers, 
mines,  rockets, 
projectiles, 
spray  tanks, 
and  mortars 

GB  and  VX 
(destroyed) 
and  mustard 
(ongoing) 

13,617 

9,292 

68 

4,325 

Operational 

Anniston 

Incineration 

Rockets  and 
projectiles 

GB 

(destroyed), 

VX  (ongoing), 
and  mustard 
(planned) 

2,254 

726 

32 

1,528 

Operational 

Umatilla 

Incineration 

Rockets, 
bombs,  and 
projectiles 

GB  (ongoing) 
and  VX  and 
mustard 
(planned) 

3,719 

1,016 

27 

2,703 

Operational 

Pine  Bluff 

Incineration 

Rockets  and 
ton  containers 

GB  (destroyed) 
and  VX  and 
mustard 
(planned) 

3,850 

484 

12 

3,366 

Operational 

Aberdeen 

Neutralization 
and  off-site 
treatment  of 
wastewater 

Ton  containers 

Mustard 

(destroyed) 

and  off-site 

wastewater 

treatment 

(complete) 

1,622 

1,622 

100 

0 

Closed 
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Site 

Destruction 

method 

Munition  types 
destroyed 

Type  of  agent 
and  status9 

Original 

agent 

tonnage 

Tonnage 

destroyed 

Percentage 
of  stockpile 
destroyed 

Remaining 
agent 
tonnage 
(as  of 
September 
2007) 

Facility 

status 

Newport 

Neutralization 
and  off-site 
treatment  of 
wastewater 

Ton  containers 

VX  (ongoing) 
and  off-site 
wastewater 
treatment 
(ongoing)0 

1,269 

587 

46 

682 

Operational 

Pueblo 

Neutralization 
and  on-site 
biotreatment 
of  wastewater 

Projectiles  and 
mortars 

Mustard 

(planned) 

2,611 

0 

0 

2,611 

Destruction 
pilot  plant 
under 

construction 

Blue 

Grass 

Neutralization 
and  on-site 
water 

oxidation  of 
wastewater 

Rockets  and 
projectiles 

GB,  VX,  and 

mustard 

(planned) 

523 

0 

0 

523 

Destruction 
pilot  plant 
under 

construction 

Total 

30,170 

14,432 

48 

15,738 

Source:  DOD. 


Note:  The  Non-Stockpile  Chemical  Materiel  Project  is  the  CMA  organization  responsible  for  all 
chemical  warfare  materiel  that  resides  outside  of  a  declared  stockpile.  This  materiel  consists  of 
chemical  agent  identification  sets,  which  consist  of  glass  ampoules,  vials  and  bottles  of  chemical 
agent,  and  actual  chemical  weapons.  The  project  provides  centralized  management  and  direction  to 
DOD  for  the  disposal  of  nonstockpile  chemical  warfare  materiel  in  a  safe,  environmentally  sound,  and 
cost-effective  manner.  It  is  responsible  for  destroying  the  nerve  agents  GB  and  VX  in  projectiles, 
vials,  bottles,  glass  ampoules,  and  ton  containers,  As  of  June  2007,  the  Non-Stockpile  Chemical 
Materiel  Project  had  destroyed  563  tons  of  chemical  agent,  which  is  counted  toward  meeting  the 
CWC  45  percent  destruction  milestone. 

“The  stockpile  includes  two  nerve  agents  (GB  and  VX)  and  a  blister  agent  (mustard). 

bThe  site  at  Johnston  Atoll  destroyed  1 ,326  tons  of  agent  before  the  CWC  entry  into  force  in  1997. 
Overall,  the  site  at  Johnston  Atoll  has  destroyed  a  total  of  2,031  tons  of  agent. 

“On  January  5,  2007,  CMA  began  shipping  Newport  VX  hydrolysate  to  a  commercial  wastewater 
treatment  facility  in  Port  Arthur,  Texas.  Because  the  VX  hydrolysate  is  being  transported  to  Texas  for 
final  treatment,  the  United  States  receives  credit  for  destruction  of  the  Newport  stockpile  under  CWC. 
Of  an  estimated  1 .8  million  gallons  of  VX  hydrolysate  at  Newport,  as  of  June  2007,  approximately  33 
percent  has  already  been  shipped  to  Texas  for  incineration. 

In  1997,  the  United  States  ratified  the  CWC,  which  prohibits  the  use  of 
these  weapons  and  mandates  the  elimination  of  existing  stockpiles  by 
April  29,  2012,  which  represents  a  5-year  extension  from  the  initial 
deadline.22  The  CWC  also  contains  a  series  of  interim  deadlines  applicable 


22The  CWC  implementing  legislation  provides  the  statutory  authority  for  domestic 
compliance  with  the  convention’s  provisions.  See  Pub.  L.  No.  105-277,  Division  I  (1998). 
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to  the  U.S.  stockpile23  for  destroying  1  percent,  20  percent,  and  45  percent 
of  the  stockpile.  The  United  States  met  the  1  percent  and  20  percent 
destruction  deadlines,  but  did  not  meet  the  original  45  percent  destruction 
deadline.  The  United  States  received  an  extension  from  April  2004  to 
December  2007  for  the  45  percent  deadline.  As  of  June  2007,  program 
officials  reported  that  45  percent  of  the  U.S  stockpile  had  been  destroyed, 

6  months  earlier  than  the  interim  CWC  deadline  for  destroying  that 
amount,  and  the  entire  stockpile  at  two  of  the  nine  sites  has  been 
eliminated. 

DOD  designated  the  Chemical  Demilitarization  Program  as  a  major 
acquisition  program.24  As  such,  the  overall  program  cost  and  schedule — 
and  its  milestones  for  major  phases  of  the  program — are  identified  in  the 
APB  and  are  reported  to  Congress  in  the  SAR.  Additionally,  the  Army, 
through  its  CMA,  developed  program  estimates  that  include  schedule 
milestones  to  support  its  day-to-day  management  responsibilities  of  the 
program.  Finally,  systems  contractors  that  operate  the  chemical  agent 
destruction  facilities  have  established  schedule  milestones  for  completing 
major  phases  identified  in  their  individual  contracts  with  the  Army. 

From  1998  through  2005,  the  total  program  cost  estimates  increased  by  113 
percent,  resulting  in  the  program  breaching  approved  program  baselines  in 
2001  and  2005.  As  a  result,  the  program  was  required  to  certify  to  Congress 
in  each  instance  that  new  cost  estimates  were  reasonable  and  that  the 
management  structure  is  adequate  to  manage  and  control  costs.  The  bulk 
of  these  program  costs  will  go  to  systems  contractors  that  construct  and 
operate  the  facilities  that  destroy  the  chemical  weapons. 


23This  report  solely  focuses  on  the  weapons  the  convention  defines  as  category  1,  which  are 
the  most  dangerous  chemicals  in  the  stockpile. 

24In  2004,  DOD  split  the  Chemical  Demilitarization  Program  into  three  major  defense 
acquisition  programs:  (1)  all  CMA  chemical  agent  destruction  sites,  except  for  the  Newport 
site;  (2)  the  Newport  site;  and  (3)  the  ACWA  program. 
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DOD  and  the  Army 
Have  Taken  Steps  to 
Strengthen  Program 
Management,  but 
Have  Not  Fully 
Developed  an 
Integrated  Approach 
for  Measuring 
Performance  and 
Managing  Risk 


DOD  and  the  Army  have  taken  steps  in  addressing  our  prior 
recommendations  to  strengthen  program  management  by  establishing  an 
overall  strategy  and  supporting  implementation  plans,  but  some  key 
elements  related  to  performance  measures  and  risk  management  are  not 
fully  developed  or  implemented.  DOD  and  the  Army  have  identified  a 
number  of  risks  across  the  program  that  could  affect  the  program’s  cost 
and  schedule,  and  CMA  has  developed  a  risk  management  plan,  but  the 
program’s  risk  management  approach  has  not  been  effective  because  it 
has  not  been  fully  implemented  or  integrated  across  the  program. 
Moreover,  while  DOD  and  the  Army  have  established  key  program  goals, 
performance  measures  have  not  always  been  linked  to  these  goals. 


DOD  Addressed  GAO’s 
Prior  Recommendations 
by  Establishing  Strategic 
and  Supporting 
Implementation  Plans 


DOD  has  taken  a  number  of  actions  to  address  our  past  recommendations 
that  it  establish  strategic  and  supporting  implementation  plans.  We 
recommended  in  2003  that  DOD  and  the  Army  develop  an  overall  strategy 
and  implementation  plan  for  the  chemical  demilitarization  program  that 
would  articulate  a  program  mission  statement,  identify  the  program’s  long¬ 
term  goals  and  objectives,  delineate  the  roles  and  responsibilities  of  all 
DOD  and  Army  offices,  and  establish  near-term  performance  measures.25 
We  also  recommended  that  DOD  and  the  Army  implement  a  risk 
management  approach  that  anticipates  and  influences  internal  and 
external  factors  that  could  adversely  affect  program  performance.  DOD 
concurred  with  our  recommendations  and  has  strengthened  program 
management  by  establishing  an  overall  strategy  and  supporting 
implementation  plans. 


In  response  to  our  2003  recommendations,  DOD  and  the  Army  have 
developed  various  strategic  and  supporting  plans.  For  example,  in  June 
2005,  DOD  published  the  Strategic  Plan  for  Destruction  of  Lethal 


zbGAO-03-1031. 
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Chemical  Agents  and  Munitions .26  The  purpose  of  this  plan  is  to  serve  as 
an  overarching  plan  for  destruction  of  the  U.S.  stockpile  of  lethal  chemical 
agents  and  munitions  and  nonstockpile  items.  It  applies  to  both  portions 
of  the  program  implemented  by  CMA  as  well  as  the  ACWA  program.  (In 
this  report,  we  only  address  the  parts  of  the  plan  pertaining  to  the  CMA 
stockpile  program.)  This  plan  included  many  of  the  elements  that  we 
recommended,  such  as  a  program  mission  statement  to  enhance  national 
security  by  eliminating  chemical  warfare  munitions  while  protecting  the 
workforce,  the  public,  and  the  environment  and  meeting  requirements 
specified  in  the  CWC.  It  also  included  overall  goals  and  objectives  and 
performance  measures. 

Also,  in  July  2005,  CMA  published  the  U.S.  Army’s  Chemical  Materials 
Agency  Stra  tegic  Plan  (2005-2010).  This  plan  encompasses  strategies  for 
all  CMA  organizational  elements  and  programs — including  the  chemical 
stockpile  storage  program,  the  Non-Stockpile  Chemical  Materiel  Project, 
the  Elimination  of  Chemical  Weapons  program,  and  the  Pine  Bluff  Arsenal 
industrial  base — together  with  those  of  the  AWCA  program.  According  to 
the  CMA  strategic  plan,  its  purpose  is  to  be  a  road  map  for  the  safe 
elimination  of  the  U.S.  chemical  weapons  stockpile.  This  strategic  plan 
articulated  program  goals  and  objectives  and  also  described  a  number  of 
supporting  implementation  plans  that  we  recommended  should  be 
developed,  including  annual  performance  plans  that  would  provide 
performance  measures  and  an  integrated  risk  management  plan  that 
would  address  the  program’s  challenges. 

In  its  July  2005  strategic  plan,  CMA  identified  11  supporting  plans  that 
were  directly  or  indirectly  related  to  the  stockpile  program.  According  to 
CMA  officials,  9  of  the  11  supporting  plans  had  been  developed,  as  of  July 
2007.  These  plans  include  the  CMA  resource  management  plan,  a  risk 
management  plan,  and  a  risk  management  directorate  plan. 

The  DOD  strategic  plan  articulated  four  key  goals  for  the  Chemical 
Demilitarization  Program:  (1)  eliminate  the  stockpile  while  protecting  the 


36Pub.  L.  No.  108-375  §  931  (2004)  directed  the  Under  Secretary  of  Defense  for  Acquisition, 
Technology,  and  Logistics  and  the  Secretary  of  the  Army  to  jointly  prepare  a  strategic  plan 
for  future  activities  for  the  destruction  of  the  U.S.  stockpile  of  lethal  chemical  agents  and 
munitions.  Section  931  further  directed  that  the  plan  include,  at  a  minimum,  realistic 
budgeting  for  stockpile  destruction  and  related  support  programs,  contingency  planning  for 
foreseeable  or  anticipated  problems,  and  a  management  approach  and  associated  actions 
that  address  compliance  with  the  obligations  of  the  United  States  under  the  CWC  treaty 
and  that  take  full  advantage  of  opportunities  to  accelerate  destruction  of  the  stockpile. 
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workforce,  the  public,  and  the  environment,  and  comply  with  U.S 
obligations  under  the  CWC;  (2)  employ  sound  business  practices  to  ensure 
efficient  life  cycle  management  of  program  resources;  (3)  institute  a 
program  of  continuous  improvement  regarding  safety,  environment 
protection,  efficient  operations  and  maintenance,  and  facility  closure  to 
mitigate  risks  and  to  ensure  compliance  with  statutory,  regulatory,  and 
policy  requirements  and  decisions;  and  (4)  maintain  communications  with 
the  public  and  local  and  national  officials.  Performance  objectives  with  the 
associated  performance  metrics  were  developed  to  address  these  goals. 
Table  2  contains  some  key  objectives  to  meet  goals  contained  in  the  DOD 
and  CMA  strategic  plans. 


Table  2:  Comparison  of  Some  Key  Destruction  Objectives  Identified  in  DOD’s  Strategic  Plan  and  CMA’s  Strategic  Plan 

DOD’s  Strategic  Plan 

CMA’s  Strategic  Plan 

Achieve  CWC  extended  45  percent  destruction  milestone  by 
December  31 , 2007,  while  ensuring  the  safety  of  the  workers,  the 
public,  and  the  environment. 

Destroy  the  U.S.  stockpile  of  lethal  chemical  agents  and  munitions 
while  meeting  the  CWC  deadlines  and  containing  total  life  cycle 
cost  by  using  an  integrated  risk  management  approach. 

Achieve  CWC  extended  100  percent  destruction  deadline  by 

April  29,  2012,  while  ensuring  the  safety  of  the  workers,  the 
public,  and  the  environment. 

No  objective  linked  to  meeting  CWC  extended  deadline  by 

April  29,  2012. 

Manage  life  cycle  costs  to  the  program  within  fiscal  resources 
and  consistent  with  the  APB. 

No  objective  linked  to  meeting  the  APB. 

Develop  and  institute  a  risk  management  culture  that  ensures 
that  no  chemical  agent  releases,  exposures,  and  accidents  occur 
across  the  entire  demilitarization  community. 

Meet  and  maintain  Army,  state,  and  federal  safety  and  security 
standards  and  environmental  laws,  regulations,  and  permit 
conditions. 

Source:  GAO  analysis  of  DOD  and  CMA  data. 


CMA  also  jointly  issued  the  program’s  Fiscal  Year  2007 Annual 
Performance  Plan  in  September  2006.27  The  performance  plan  contains, 
among  other  things,  annual  goals  for  meeting  the  CWC’s  December  2007 
interim  destruction  deadline.  However,  Tooele  is  the  only  site  that  has 
approved  annual  plans,  which  it  is  implementing.  CMA  officials  stated  that 
they  will  request  annual  performance  plans  from  all  of  the  sites  for  fiscal 
year  2008.  Moreover,  the  plans  lacked  some  attributes  that  would  make 
them  more  useful  to  decision  makers  and  to  Congress.  We  have  previously 
found  that  performance  plans  can  be  more  effective  if  they  (1)  use 
intermediate  goals  and  measures  to  show  progress  or  contribution  to 
intended  results,  particularly  if  it  may  take  years  to  meet  the  intended 


"  The  performance  plan  applies  to  both  CMA  and  the  ACWA  portions  of  the  demilitarization 
program.  It  was  the  program’s  first  plan  and  was  issued  jointly  by  both  organizations. 
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results;  (2)  show  baseline  and  trend  data  for  past  performance;  and 
(3)  permit  an  agency  to  convey  what  it  expects  to  achieve  in  the  long  term 
by  including  multiyear  performance  goals  in  its  performance  plan.28  With 
baseline  and  trend  data,  the  more  useful  performance  plans  provided  a 
context  for  drawing  conclusions  about  whether  performance  goals  are 
reasonable  and  appropriate.  By  including  multiyear  performance  goals,  a 
performance  plan  can  provide  congressional  and  other  decision  makers 
with  an  indication  of  the  incremental  progress  the  agency  expects  to  make 
in  achieving  results.  CMA’s  performance  plan  only  contains  goals  related 
to  the  specified  fiscal  year  and  does  not  show  previous  years’  trends  or 
show  how  achievement  of  the  specified  goals  would  contribute  to  the 
program’s  long-term  goal. 


Risk  Management 
Approach  Is  Not  Fully 
Developed  or  Integrated, 
and  Some  Key  Program 
Goals  Lack  Clearly  Linked 
Performance  Measures 


DOD  has  partially  implemented  our  recommendations  to  establish 
performance  measures  and  implement  a  risk  management  approach  that 
anticipates  and  influences  internal  and  external  factors  that  could 
adversely  affect  program  performance.  Specifically,  CMA  has  not  yet 
completed  all  the  risk  management  tools  identified  in  its  risk  management 
plan  or  integrated  its  risk  management  efforts  across  sites  and 
organizations.  Further,  while  DOD  did  establish  performance  measures  as 
we  recommended,  some  key  program  goals  still  lack  clearly  linked 
performance  measures.  Moreover,  the  program  is  using  four  different 
performance  measurement  approaches  and  measurements  that  are  not 
well-aligned  with  one  another.  Without  annual  performance  goals  and 
measures  linked  with  long-term  strategic  goals,  program  officials  will  lack 
information  on  the  interim  progress  being  achieved  and  whether 
corrective  action  is  needed  to  safely  destroy  the  stockpile  by  the  treaty 
deadline. 


28GAO,  Agency  Performance  Plans:  Examples  of  Practices  That  Can  Improve  Usefulness 
to  Decisionmakers,  GAO/GGD/AIMD-99-69  (Washington,  D.C.:  Feb.  26,  1999). 
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CMA  Has  Made  Progress  in 
Developing  an  Integrated  Risk 
Management  Approach  but  Has 
Not  Completed  Its  Risk 
Management  Tools  and 
Integration  Efforts 


As  identified  in  its  U.S.  Army  Chemical  Materials  Agency  Integrated  Risk 
Management  Plan,  CMA  has  developed  a  process  to  identify  program-level 
risks  and  reports  them  as  part  of  the  Army’s  Probability  of  Program 
Success  Report.29  The  program  office,  in  conjunction  with  the  site  project 
managers,  also  has  developed  a  “risk  landscape,”  which  is  used  to  capture 
risks  that  could  affect  the  operating  and  closure  schedules.  However,  no 
programwide  approach  exists  to  integrate  the  risk  management  efforts  at 
the  site,  CMA,  and  DOD  levels  to  ensure  that  all  risks  are  being  addressed 
in  an  integrated  manner.  For  example,  CMA  officials  acknowledged  that 
risks  that  are  included  in  the  Probability  of  Program  Success  Reports  and 
the  site  risk  landscapes  were  developed  independently  of  each  other  and 
thus  may  not  be  fully  integrated.  Also,  while  the  sites  were  developing  risk 
plans  that  are  at  different  stages  of  maturity,  CMA  has  not  yet  developed  a 
single  set  of  risk  management  procedures  to  be  used  by  the  site-level 
project  managers  and  site  systems  contractors  when  developing  these 
plans.  Lastly,  the  risks  identified  in  DOD  plans  and  those  identified  by 
CMA  were  not  identified  as  part  of  an  integrated  process.  CMA  officials 
stated  that  the  risks  it  identified  are  an  aggregate  of  some  site  risks  and 
CMA  program-level  risks,  whereas  the  risk  identified  by  DOD  in  its 
strategic  plan  represents  a  general  analysis  of  the  major  factors  that  the 
senior  leadership  has  selected  as  appropriate  for  long-term  attention.  CMA 
officials  stated  that  CMA  is  working  to  establish  a  formal  process  for 
aligning  risks  among  all  planning  and  reporting  documents. 

Other  steps  that  are  needed  to  fully  implement  this  approach  have  not 
been  completed.  For  instance,  CMA  identified  other  tools  to  aid  in 
implementing  an  overarching  risk  management  program  to  complete  the 
steps  in  this  process,  but  these  tools  have  not  been  fully  developed.  These 
tools  include  a  risk  trade-off  analysis  and  a  risk  management  information 
system  database.  A  risk  trade-off  analysis  is  normally  included  in  this  risk 
planning  step.  According  to  CMA  officials,  while  risk  trade-off  analyses  are 
discussed  in  their  risk  management  plan,  the  program  has  not  formally 
used  this  analysis  in  assessing  program  risk,  and  thus  this  analysis  has  not 
been  available  to  help  decision  makers  consider  how  to  address  program 
challenges.  For  example,  while  the  condition  of  the  stockpile  of  mustard 
agent  at  Tooele  was  identified  as  a  key  program  risk,  a  process  was  not  in 
place  to  perform  a  comprehensive  analysis  to  consider  the  risks  to  the 


29The  Probability  of  Program  Success  Report  is  designed  to  improve  the  Army’s  ability  to 
accurately  assess  the  program’s  probability  of  success,  and  clearly  and  concisely  represent 
that  success  probability  to  leadership. 
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Some  Key  Program  Goals  Lack 
Clearly  Linked  Performance 
Measures 


program’s  cost  and  schedule  if  assumptions  about  the  level  of  mercury 
contamination  were  understated.  Because  the  extent  of  the  problem  was 
not  discovered  sooner,  needed  facility  modifications  were  delayed, 
extending  the  planned  operations  schedule  by  12  months. 

CMA  has  also  not  completed  the  development  of  its  integrated  risk 
information  management  system,  which  would  involve  the  collection, 
documentation,  and  reporting  of  risk  data  in  support  of  the  steps  in  the 
overall  risk  management  process.  According  to  CMA,  an  initial 
requirement  analysis  for  the  development  of  an  integrated  risk  information 
system  determined  that  initially  considered  in-house  solutions  would 
require  considerable  additional  development.  A  market  survey  will  be 
conducted  to  consider  external  solutions.  While  a  comprehensive  risk 
management  approach  could  manage  risks,  and  CMA  is  still  in  the  process 
of  developing  such  an  approach,  CMA  has  not  established  time  frames  for 
the  completion  and  implementation  of  its  risk  management  approach. 
Without  clear  expectations  for  fully  developing  and  implementing  its  risk 
management  approach  within  a  set  l  ime  frame,  CMA  could  continue  to  be 
limited  in  its  efforts  to  control  some  of  the  issues  that  have  been  affecting 
the  program. 

While  DOD  and  the  Army  have  made  progress  in  establishing  program 
goals,  we  found  that  some  program  goals  were  not  linked  to  one  another 
or  to  key  performance  measures,  and  making  such  linkages  is  a  practice  of 
leading  organizations.30  For  example,  in  its  fiscal  year  2007  annual 
performance  plan,  CMA  had  developed  an  annual  performance  measure 
linked  to  the  interim  CWC  destruction  goal  of  45  percent  by  December 
2007.  However,  neither  DOD  nor  CMA  nor  the  Army  has  developed  annual 
performance  measures  that  are  linked  to  achieving  the  ultimate  goal  of 
destroying  the  total  stockpile  at  each  remaining  site  by  the  April  29,  2012, 
treaty  deadline.  Additionally,  while  the  Army  has  recently  established 
annual  site  goals  for  chemical  agent  tons  destroyed,  these  targets  have 
been  linked  either  to  Army  or  DOD  schedules  that  far  exceed  the  extended 
April  29,  2012,  treaty  deadline.  Without  this  linkage,  it  is  difficult  for 
program  officials  to  track  the  progress  that  the  program  is  making  toward 
its  goals,  and  program  officials  have  little  assurance  that  program 
activities  are  linked  to  program  goals. 


30See,  for  instance,  GAO,  Executive  Guide:  Effectively  Implementing  the  Government 
Performance  Results  Act,  GAO/GGD-96-118  (Washington,  D.C.:  June  1996). 
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The  program  is  using  four  different  performance  measurement 
approaches:  the  SAR,  the  Office  of  Management  and  Budget’s  (OMB) 
Program  Assessment  Rating  Tool  (PART),31  and  DOD’s  and  CMA’s 
strategic  plans.  As  this  is  a  major  acquisition  program,  some  performance 
measures  related  to  cost  and  schedule  are  contained  in  the  SAR.  However, 
while  DOD  and  CMA  officials  cited  the  schedule  milestones  in  the  APB  as 
the  schedule  the  program  is  accountable  to,  neither  the  2006  approved 
APB,  the  April  2006  SAR,  nor  the  April  2007  SAR  contained  schedule 
milestone  dates  beyond  April  2012,  but  rather  stated  that  schedule 
milestones  are  “to  be  determined.”  DOD  officials  stated  that  the  dates  are 
not  included  in  the  APB  because  it  was  approved  about  1  week  before 
DOD  officially  announced  to  Congress  that  all  sites  would  not  meet  the 
extended  April  29,  2012,  treaty  deadline.  Although  18  months  (as  of 
October  2007)  have  elapsed  since  DOD  made  that  announcement,  the 
dates  have  still  not  been  included  in  these  documents.  DOD  and  CMA 
officials  stated  that  the  milestone  dates  are  still  being  used  for 
performance  tracking  despite  not  being  included  in  the  most  recent  SARs. 
While  the  program  may  have  a  measurement  for  tracking  progress,  its 
exclusion  from  the  SAR  limits  program  oversight  by  Congress. 

The  program  also  developed  additional  performance  measures  beyond 
those  used  in  the  acquisition  reporting  system.  For  example,  the  program 
developed  performance  measures  in  (1)  the  Office  of  the  Secretary  of 
Defense’s  (OSD)  strategic  plan,  (2)  OMB’s  PART,  and  (3)  CMA’s  strategic 
plan.  However,  these  measures  have  not  always  been  linked  to  each  other 
and  have  not  always  been  linked  to  key  program  goals.  For  example,  while 
the  achievement  of  destruction  of  the  stockpile  by  the  treaty  deadline  is  an 
overall  program  goal  in  both  the  DOD  and  CMA  strategic  planning 
documents,  neither  DOD  nor  CMA  has  developed  annual  performance 
measures  that  are  linked  to  achieving  that  goal  even  though  they  had 
established  goals  linked  to  the  interim  45  percent  destruction  goal. 


31OMB’s  PART  was  developed  to  assess  and  improve  program  performance  so  that  the 
federal  government  can  achieve  better  results.  A  review  using  this  tool  helps  identify  a 
program’s  strengths  and  weaknesses  to  inform  funding  and  management  decisions  aimed 
at  making  the  program  more  effective.  The  tool,  therefore,  looks  at  all  factors  that  affect 
and  reflect  program  performance,  including  program  purpose  and  design;  performance 
measurement,  evaluations,  and  strategic  planning;  program  management;  and  program 
results.  Because  the  tool  includes  a  consistent  series  of  analytical  questions,  it  allows 
programs  to  show  improvements  over  time,  and  allows  comparisons  between  similar 
programs.  See  Office  of  Management  and  Budget,  Program  Assessment  Rating  Tool 
(PART),  http://www.whitehouse.gov/omb/part/. 
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Leading  organizations  seek  to  establish  clear  hierarchies  of  performance 
goals  and  measures  and  link  these  goals  and  measures  for  each  successive 
organizational  level  to  the  overall  program  goal.32  Our  analysis  of  the 
various  performance  measures  also  determined  that  they  were  not  always 
linked  to  one  another  or  tied  to  key  program  goals.  For  example,  while 
OSD  identified  14  performance  measures  that  support  the  four  goals 
contained  in  the  strategic  plan  that  directly  relate  to  the  Chemical 
Demilitarization  Program’s  stockpile  program,  CMA’s  performance 
measures  are  not  always  aligned  with  OSD’s  performance  measures.  Also, 
while  OSD  identified  performance  measures  that  are  similar  to  those 
tracked  by  OMB  in  the  PART,  two  performance  measures  in  the  PART  are 
not  included  in  OSD’s  strategic  plan.  These  two  measures  were  the 
planned  cost  per  ton  of  agent  destroyed  against  actual  cost  and  the 
statistical  confidence  of  meeting  the  45  percent  CWC  deadline.  The  SAR 
contained  performance  measures  related  to  safety  and  environmental 
performance,  such  as  those  related  to  chemical  exposures  and  releases, 
but  did  not  have  performance  measures  related  CWC  tonnage  goals.  Table 
3  compares  measures  identified  in  the  PART,  the  OSD  strategic  plan,  the 
CMA  strategic  plan,  and  the  SAR. 


32GAO/GGD/AIMD-99-69. 
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Table  3:  Comparison  of  Performance  Measures  Identified  in  the  PART,  OSD  Strategic  Plan,  CMA  Strategic  Plan,  and  Selected 
Acquisition  Report 

OMB  PART  measures 

OSD  strategic  plan 

CMA  strategic  plan 

SAR  (CMA  and  Newport) 

Number  of  chemical 
agent  exposures 

No  chemical  agent  exposures  as 
defined  in  the  APB 

Number  of  chemical  exposures  as 
defined  in  the  APB 

Number  of  chemical  agent 
exposures 

Number  of  chemical 
agent  releases 

No  chemical  releases  as  defined  in 
the  APB 

Number  of  chemical  agent  releases 
as  defined  in  the  APB 

Number  of  chemical  agent 
releases 

Tons  of  chemical  agent 
destroyed  against  CWC 
45  percent  destruction 
deadline 

Percentage  of  actual  amount  of 
stockpiled  chemical  materiel 
destroyed  against  CWC  45  percent 
destruction  deadline 

No  tonnage  destroyed  performance 
measure  against  CWC  45  percent 
destruction  deadline 

No  CWC  tonnage  destroyed 
performance  measure 

Tons  of  chemical  agent 
destroyed  against  CWC 
100  percent  destruction 
deadline 

Percentage  of  actual  amount  of 
stockpiled  chemical  materiel 
destroyed  against  CWC  100  percent 
destruction  deadline 

No  tonnage  destroyed  performance 
measure  against  CWC  100  percent 
destruction  deadline 

No  CWC  tonnage  destroyed 
performance  measure 

Recordable  incidence 
rate 

Recordable  incidence  rate 

Recordable  incidence  rate 

Meet  Army,  state,  and/or 
federal  requirements 

Percentage  of  stockpile 
destroyed  against  CWC 
45  percent  destruction 
deadline 

No  related  measure 

Statistical  confidence  in  achieving 
CWC  45  percent  destruction  deadline 

No  related  measure 

Planned  cost  per  ton  of 
agent  destroyed  against 
actual  cost 

No  related  measure 

Plan  cost  per  ton  of  agent  destroyed 
against  actual  cost 

No  related  measure 

Source:  GAO  analysis  of  DOD  and  CMA  data. 


This  lack  of  integration  between  key  program  goals  and  performance 
measures  hampers  program  officials’  ability  to  effectively  track  the 
progress  that  the  program  is  making  because  the  performance  plan  lacks 
destruction  goals,  approaches,  and  milestones  that  are  directly  linked  to 
overall  program  goals  for  safely  meeting  CWC  deadlines. 
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Recently  Improved 
Destruction  Rates  for 
Chemical  Munitions 
May  Indicate  Overly 
Conservative 
Schedule  Milestones 
at  the  Four 
Incineration  Sites 


Since  DOD  extended  the  program  office  schedules  and  DOD  schedule 
milestones  in  2005,  actual  processing  rates  have  considerably  exceeded 
the  forecasted  rates  at  the  four  incineration  sites,  indicating  that  schedule 
milestones  being  used  to  measure  the  program’s  progress  may  be  too 
conservative  to  be  realistic.  DOD’s  Chemical  Demilitarization  Program 
schedule  has  been  extended  in  recent  years  largely  because  of  slower- 
than-expected  processing  rates  for  destroying  chemical  agent-filled 
munitions.  Program  office  schedules  were  extended  in  2005  because  of  a 
greater  recognition  of  slower  processing  rates  for  destroying  chemical 
munitions  than  schedule  milestones  adopted  in  the  more  optimistic  2003 
APB.  In  addition,  the  program  office  employed  a  new  schedule  forecasting 
methodology  that  also  contributed  to  the  schedule  extensions  because  it 
accounted  for  more  equipment  downtime  and  for  the  risks  that  have 
delayed  processing  in  the  past.  The  schedule  extensions  exceeded  the 
existing  2003  baseline  and  led  DOD  to  adopt  new  schedule  milestones  in 
2006.  Finally,  program  confidence  levels  for  meeting  the  April  29,  2012, 
CWC  milestone  are  low. 


Slower-Than-Anticipated 
Processing  Rates  and  Time 
Frames  Led  to  Schedule 
Extensions 


The  program  office  schedules  have  been  extended  since  2003  largely 
because  forecasted  processing  rates  for  destroying  chemical  agent-filled 
munitions  were  reduced  based  on  the  program’s  analysis  of  historical 
processing  rates  being  slower  than  anticipated.  Our  review  of  best 
practices  indicates  that  estimating  assumptions  should  be  realistic  and 
backed  by  historical  data  to  minimize  uncertainty  and  risk.33  Best  practices 
also  stipulate  that  estimates  are  accurate  when  they  are  not  overly 
conservative  or  optimistic.  Best  practices  also  recommend  that  when 
underlying  data  or  other  assumptions  change,  cost  and  schedule  estimates 
should  be  revised  to  reflect  the  current  status.  According  to  program 
officials,  previous  program  office  schedule  estimates  developed  in  the 
early  2000s  were  based  on  assumptions  about  processing  rates  derived 
largely  from  peak  processing  experiences  at  Johnston  Atoll,  which  began 
operations  in  1990,  and  early  munitions  processing  campaigns  at  Tooele — 
the  first  two  sites  in  operation.  According  to  CMA  officials,  program  office 
schedule  estimates  used  from  2001  to  2003  were  derived  from  using  the  5 
best  weeks  of  these  two  sites’  production  rates.  The  program’s  assumption 
was  that  the  follow-on  sites,  which  were  similarly  designed,  would  be  able 
to  achieve  these  rates  because  of  the  program’s  years  of  experience  with 
the  various  munitions  and  agents.  Program  officials  said  that  in  practice 


33GAO-07-1 134SP. 
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those  rates  were  achievable  only  for  short  periods  and  were  not 
sustainable  because  of  problems  with  individual  munitions,  such  as 
difficulty  in  draining  the  agent  (much  slower  than  assumed),  equipment 
jams,  and  processing  restrictions  specified  in  facility  operating  permits. 


New  Schedule 
Methodology  Includes 
More  Time  for  Downtime 
and  Processing  Risks 


The  program  office  adopted  a  new  schedule  development  methodology  in 
2004  that  not  only  used  modeling  to  determine  processing  rates,  but  gave 
greater  recognition  to  the  issues  that  cause  delays,  such  as  plant  downtime 
days  for  equipment  maintenance  and  the  inclusion  of  risks,  such  as  those 
associated  with  the  unexpected  condition  of  the  stored  munitions  that 
further  extended  the  schedule.  While  previous  schedule  estimates  were 
point  estimates,  in  that  a  single  processing  rate  was  developed  and  used 
for  each  campaign  and  the  individual  campaigns  were  added  together, 
statistical  modeling  was  used  to  develop  a  distribution  of  different 
schedule  outcomes.  Processing  rates  were  evaluated  by  munitions  type, 
such  as  projectiles,  rockets,  and  mines,  to  develop  a  histogram  of  daily 
processing  outcomes.34  These  results  were  used  as  part  of  the  modeling 
inputs.  For  current  estimates,  unlike  past  estimates,  schedule  estimators 
also  collected  data  on  issues  that  had  caused  processing  delays  in  the  past 
and  identified  possible  impacts  on  the  schedule  and  probabilities  of 
occurrence. 


Data  were  also  collected  on  past  program  incidents  that  had  caused  sites 
to  remain  idle  for  extended  periods,  and  the  estimators  also  developed 
probabilities  for  these  “downtime  events.”  The  model  was  run  to  produce 
a  distribution  of  5,000  schedule  outcomes.  The  median  result — referred  to 
as  the  50  percent  confidence  level — was  selected  as  the  program  office 
schedule  estimate.  Selection  of  the  median  result  was  also  considered 
conservative,  since  the  previous  program  office  schedules  have  used  the 
mode,  which  is  the  outcome  that  happens  most  frequently.  According  to 
schedule  estimators,  the  median  result  is  typically  to  the  right  of  or  longer 
than  the  mode. 


Thus,  when  the  schedule  estimates  were  updated  in  2005  using  the  median 
schedule  outcomes,  they  were  not  only  more  conservative,  but  also 
considered  more  realistic.  Figure  5  shows  how  processing  rates, 
downtime,  and  risk  contributed  to  the  schedule  estimate. 


34A  histogram  is  a  graphical  display  of  tabulated  frequencies. 
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Figure  5:  2005  Program  Office  Schedule  Estimates  by  Processing  Rates,  Downtime,  and  Risk 
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Source:  CMA. 


Program  Office  Schedule  When  these  schedule  estimates  exceeded  the  schedule  milestones 
Extensions  Led  to  Revised  contained  in  the  April  2003  APB  and  resulted  in  major  changes  to  the  life 
Program  Baseline  cycle  cost  estimates,  an  acquisition-related  process  was  initiated  to 

develop  new  schedule  milestones.  The  new  schedule  estimate  was 
approved  by  the  Army  Cost  Review  Board  in  May  2005  and  resulted  in  the 
new  Army  Cost  Position.  Concurrently,  DOD  was  also  conducting  a 
program  review  through  its  Defense  Acquisition  Board  process  that  would 
result  in  new  schedule  estimates  to  be  included  in  the  program’s  Approved 
Acquisition  Program  Baseline.  As  part  of  this  process,  the  DOD’s  Cost 
Analysis  Improvement  Group  (CAIG)  developed  independent  schedule 
estimates.36  CAIG’s  schedule  forecasts  were  also  based  on  modeling  to 
select  the  median  outcome.  Unlike  those  based  on  the  program 
rebaselining  that  occurred  from  2001  to  2003,  the  program  office  schedule 
estimates  and  those  developed  by  CAIG  were  relatively  close  in  most 


35The  OSD  CAIG  has  specific  responsibility  for  providing  independent  cost  estimates  when 
requested  by  the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  in 
support  of  major  milestone  decision  points  (Milestone  B,  Milestone  C,  or  the  full-rate 
production  decision  review).  Overall,  CAIG  serves  as  the  principal  advisory  body  to  the 
Milestone  Decision  Authority  on  all  matters  concerning  an  acquisition  program’s  life  cycle 
cost. 
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cases.  The  schedule  milestones  in  the  2003  APB  are  compared  with  those 
in  the  2005  Army  and  DOD  schedule  estimates  in  table  4. 


Table  4:  Comparison  of  Estimated  Operation  Completion  Dates 


Site 

2003  Acquisition 
Program 

Baseline 

2005  Army 
program  office 
estimate 

Difference  from 
2003  Acquisition 
Program 

Baseline 

2005  DOD 

independent 

estimate 

Difference  from 
2003  Acquisition 
Program 

Baseline 

Program  current 
estimate 

Anniston 

May  201 1 

September  2016 

64  months 

June  2016 

61  months 

May  2015 

Newport 

July  2005 

October  201 1 

75  months 

February  2012 

73  months 

October  2009 

Pine  Bluff 

November  2009 

January  2015 

62  months 

November  2015 

72  months 

June  2013 

Tooele 

February  2008 

October  2014 

80  months 

September  2015 

91  months 

November  2014 

Umatilla 

January  201 1 

June  2017 

77  months 

March  2017 

74  months 

December  2014 

Source:  GAO  analysis  of  DOD  and  CMA  data. 


Recently  Improved 
Processing  Rates  May 
Make  the  Current  Schedule 
Milestones  Overly 
Conservative,  but  Program 
Confidence  Levels  for 
Meeting  Treaty  Milestones 
Are  Low 


Our  review  of  processing  rates  that  have  been  achieved  at  the  sites  since 
the  program  office  estimates  were  developed  in  2005  determined  that  most 
sites  are  significantly  ahead  of  the  program  office  estimate  for  recently 
completed  munition  campaigns,  indicating  that  schedule  milestones  being 
used  to  measure  the  program’s  progress  may  be  too  conservative  to  be 
realistic.  For  example,  the  2005  program  office  schedule  forecasted  that 
Umatilla  would  complete  destruction  of  its  91,442  sarin  nerve  agent  rocket 
munitions  by  August  2007.  Those  operations  were  actually  completed  in 
August  2006 — about  12  months  ahead  of  the  program  office  2005  estimate. 
This  processing  rate  was  achieved  despite  delays  to  the  site  operator 
internal  schedule  related  to  the  unanticipated  fires  associated  with  some 
of  the  rockets.  Additional  progress  made  through  May  2007  has  placed 
Umatilla  about  1,184  days  ahead  of  2005  program  office  schedule  estimate. 
We  found  similar  occurrences  at  the  other  incineration  sites,  and  actual 
and  near-term  projected  performance  by  sites  reveals  that  significant 
schedule  progress  has  been  made  since  the  2005  schedule-estimating 
process.  Table  5  shows  actual  and  projected  schedule  savings  as 
compared  to  the  2005  program  office  estimated  schedule  for  campaigns 
recently  completed  or  under  way. 
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Table  5:  Number  of  Days  Saved  per  Site  since  2005  Program  Office  Schedule  Estimates 


Site 

Cumulative  days 
saved 

Actual  or  projected 
days  saved 

End  date  of  last 
completed  campaign 
completed  to  date 

Campaigns  ongoing  and  completed  or 
started  and  completed  from  May  2005 
through  July  2007 

Anniston 

443 

Actual 

March  8,  2007 

GB  8-inch  projectiles 

GB  155-mm  projectiles 

GB  M360  105-mm  projectiles 

VX  M55  rockets/warheads 

Newport 

658 

Projected 

December  2,  2009a 

VX  agent  neutralization 

Pine  Bluff 

375 

Actual 

May  19,  2007 

GB  M55  rockets/warheads 

Tooele 

61 

Actual 

August  17,  2006 

VX  land  mines 

Agent  changeover 

Umatilla 

1,184 

Actual 

January  24,  2007 

GB  M55  rockets/warheads 

GB  MC-1  bombs 

GB  MK-166  bombs 

GB  8-inch  projectiles 

GB  155-mm  projectiles 

Aberdeen 

750 

Actual 

February  13,  2006 

Ton  container  cleanout 

Total 

3,471 

Source:  GAO  analysis  of  Army  data. 

“Figures  listed  above  for  Anniston,  Pine  Bluff,  Tooele,  and  Umatilla  represent  actual  savings.  Figures 
for  Newport  are  projected  savings  for  current  ongoing  campaigns  based  on  the  historical  processing 
rates  at  those  sites.  The  Newport  forecast  for  completing  the  current  campaign  is  December  2009. 

We  calculate  that  to  date,  almost  $1.3  billion  in  cost  savings  has  accrued 
based  on  schedule  savings  since  the  2005  program  office  estimates  were 
established.  Table  6  shows  actual  and  projected  schedule  savings  as 
compared  to  the  2005  estimated  schedule  savings  for  campaigns  recently 
completed  or  under  way. 
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Table  6:  Potential  Operational  Cost  Savings  Associated  with  Days  Saved  since  2005 
Schedule  Estimate 

Dollars  in  millions 

Site 

Cumulative  days  saved 

Potential  cost  savings 

Anniston 

443 

$178 

Newport 

658 

227 a 

Pine  Bluff 

375 

144 

Tooele 

61 

25 

Umatilla 

1,184 

452 

Aberdeen 

750 

236 

Total 

3,471 

$1,262 

Source:  GAO  analysis  of  Army  data. 


Note:  Time  savings  were  converted  to  a  cost  by  using  the  average  fiscal  year  2005  and  fiscal  year 
2006  budgets  expressed  in  dollars  per  day. 

“Figures  listed  above  for  Anniston,  Pine  Bluff,  Tooele,  and  Umatilla  represent  actual  savings.  Figures 
for  Newport  are  projected  savings  for  current  ongoing  campaigns  based  on  the  historical  processing 
rates  at  those  sites.  The  Newport  forecast  for  completing  the  current  campaign  is  December  2009. 

The  April  2007  SAR  for  the  CMA  sites  (including  Newport)  reported  new 
program  office  schedule  estimates,  which  were  a  significant  improvement 
from  the  Army’s  2005  program  office  schedule  estimates  that  generated 
the  program’s  rebaselining  and  also  were  significantly  ahead  of  the 
independent  DOD  schedule  estimates  that  are  used  as  the  program’s 
unofficial  schedule  baseline  and  were  developed  by  CAIG.  Table  7  shows 
the  number  of  months  that  the  estimated  schedule  for  completing 
operations  has  improved  from  2005  to  2007. 
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Table  7:  Number  of  Months  of  Improvement  in  2007  Program  Current  Estimate  for  Operations  Completion  versus  2005  Army 
Program  Office  Estimate  and  2005  DOD  Independent  Estimate 

Site 

2005  Army 
program  office 
estimate 

2005  DOD 

independent  estimate 

2007  program 
current  estimate 

Improvement  from 
2005  program  office 
estimate 

Improvement  from 
2005  independent 

DOD  estimate 

Anniston 

September  2016 

June  2016 

May  2015 

16  months 

13  months 

Newport 

October  201 1 

February  2012 

October  2009 

24  months 

28  months 

Pine  Bluff 

January  2015 

November  2015 

June  2013 

19  months 

29  months 

Tooele 

October  2014 

September  2015 

November  2014 

(1  month) 

10  months 

Umatilla 

June  2017 

March  2017 

December  2014 

30  months 

27  months 

Source:  GAO  analysis  of  DOD  and  CMA  data. 


In  addition  to  its  annual  schedule  update  process,  CMA  tracks  on  a 
monthly  basis  the  confidence  levels  of  achieving  various  program  schedule 
estimates.  CMA  tracks  these  levels  after  updating  its  model  to  incorporate 
updated  processing  rates  and  any  changes  to  estimates  for  downtime  or 
risk.  These  updates  have  shown  upward  trends  in  the  confidence  level, 
which  is  the  percentage  of  schedule  outcomes  that  meet  or  exceed  a  given 
date.  As  previously  discussed,  the  2005  program  office  estimate  reflected  a 
50  percent  confidence  level.  By  the  end  of  fiscal  year  2006,  which  was 
about  18  months  after  the  2005  program  office  estimates  were  developed, 
only  Tooele,  at  48  percent,  was  slightly  below  the  50  percent  confidence 
level.  The  other  three  sites’  confidence  levels  for  achieving  the  2005 
program  office  estimate  for  completing  operations  had  increased 
significantly,  ranging  from  71  percent  at  Anniston  to  83  percent  at 
Umatilla.  Moreover,  confidence  levels  based  on  monthly  updates 
conducted  in  April  2007  showed  that  the  program  office  estimates  for 
meeting  the  DOD  schedule  milestones  were  as  high  as  98  percent  for 
completing  operations,  but  confidence  levels  were  very  low  for  meeting 
the  April  29,  2012,  treaty  deadline,  as  shown  in  table  8. 
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Table  8:  CMA  Assessed  Confidence  in  Achieving  DOD  Schedule  Milestone  and 
April  29,  2012,  Treaty  Deadline 


Percentages 


Site 

Program’s  assessed 
confidence  in  achieving 
operations  schedule  milestone 

Program’s  assessed  confidence  in 
sites  completing  destruction  by 
April  29,  2012,  CWC  milestone 

Anniston 

73 

5 

Newport 

98 

98 

Pine  Bluff 

81 

19 

Tooele 

42 

1 

Umatilla 

84 

2 

Source:  CMA. 

While  the  program  office  has  updated  its  schedule  estimates  to  reflect  the 
latest  processing  rates  and  changes  to  other  assumptions,  resulting  in 
schedule  improvements  in  excess  of  18  months  in  most  cases,  independent 
schedule  estimates  have  not  been  developed  since  the  CAIG  2005 
estimates  because  these  estimates  are  usually  performed  in  conjunction 
with  program  rebaselinings  because  of  schedule  or  cost  growth.  Also, 
when  schedules  or  other  assumptions  change,  cost  and  schedule  estimates 
should  be  revised  to  reflect  current  status.  Because  these  steps  have  not 
been  taken,  DOD  and  Congress  can  have  little  confidence  that  the  program 
is  being  held  accountable  to  a  reasonable  baseline  to  measure  program 
progress  and  that  program  costs  are  reasonable. 


Shortcomings  in 
Underlying  Cost  Data 
Make  Earned  Value 
Management  Results 
Unreliable  and 
Limited  Our  Ability  to 
Determine  Cost 
Estimate  Reliability 


Because  of  shortcomings  in  the  underlying  data,  the  usefulness  of  the  EVM 
data  to  make  cost  projections  is  limited,  which  makes  results  unreliable, 
and  our  ability  to  independently  verify  and  validate  the  reliability  of  the 
cost  estimates  was  restricted.  The  Chemical  Demilitarization  Program’s 
projected  cost  growth  is  largely  attributable  to  longer  schedules  and 
increased  costs  associated  with  facility  closure  estimates.  Shortcomings  in 
the  underlying  data  include  invalid  and  unstable  performance  baselines. 
The  performance  baselines  are  problematic  because  all  requirements,  such 
as  closure  costs,  are  not  fully  identified  and  included  in  program  cost 
estimates  and  because  large  amounts  of  additional  costs  are  added  to  the 
performance  baseline  annually.  Moreover,  there  were  significant 
differences  between  the  systems  contractor  cost  estimates  through 
completion  and  the  government  cost  estimates  through  completion 
because  the  government  anticipates  longer  operational  periods  and 
greater  closure  requirements. 
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Projected  Cost  Growth  Is 
Largely  Attributable  to 
Longer  Schedules  and 
Increased  Facility  Closure 
Estimates 


The  Chemical  Demilitarization  Program’s  continuing  growth  in  cost 
estimates  is  largely  attributable  to  longer  schedules  and  the  increased 
costs  associated  with  plant  closure.  Longer  schedules  led  to  increases  in 
cost  estimates  for  sites  currently  in  the  operations  phase  because  of 
additional  labor  costs,  which  are  a  high  percentage  of  site  costs.  In 
addition,  according  to  CMA  officials,  labor  costs  are  very  expensive  for 
this  program  because  of  the  need  for  highly  specialized  labor  that  cannot 
be  easily  replaced.  According  to  these  officials,  each  staff  member,  for 
example,  must  pass  the  Personal  Reliability  Program,  which  provides 
specialized  training  and  security  clearances  for  working  at  the  chemical 
deconstruction  plants.  Program  officials  have  stated  that  for  this  reason,  it 
is  often  more  cost-effective  during  plant  shutdowns  or  extended  delays  to 
retain  this  specialized  workforce  than  to  lay  off  and  rehire  new  staff.  This 
workforce  costs  approximately  $9.5  million  per  month  based  on  an 
average  monthly  spend  rate  across  the  five  sites  we  examined,  which 
means  that  a  1-month  schedule  extension  could  increase  the  life  cycle  cost 
estimates  by  about  that  amount.  Table  9  shows  the  increase  in  program 
cost  as  the  schedule  was  extended. 


Table  9:  Changes  in  Estimated  Total  Program  Cost 


Then-year  dollars  in  billions 

Source  of  2005  Army  cost  April  2007 

estimate  2003  APB  position  Increase  SAR 

Total  life  cycle  cost  $19.6  $29.2  $9.6  $28.6 

estimate 


Source:  U.S.  Army. 


While  the  Chemical 
Demilitarization  Program 
Is  Using  Earned  Value 
Management  as  a  Tool, 
Some  Underlying  Data 
Make  Results  Unreliable 


EVM  is  an  important  program  management  tool  required  by  DOD  and 
OMB  that  measures  performance  by  comparing  the  value  of  work 
accomplished  with  the  value  of  work  planned  and  the  cost  of  the  work 
performed.  Using  EVM  data,  early  warning  of  schedule  delays  and  cost 
overruns  can  be  determined.  EVM  uses  contractor-reported  data  to 
examine  variances  reported  in  contractor  performance  reports  between 
actual  cost  and  time  of  performing  work  tasks  and  the  budgeted  or 
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estimated  cost  and  time.  As  a  result,  DOD  and  OMB  policy  and  guidance 
require  the  use  of  EVM  to  measure  program  performance.36 

The  Chemical  Demilitarization  Program  uses  EVM37  as  a  tool  that  allows 
both  government  and  systems  contractor  managers  to  have  visibility  over 
cost  performance  on  their  contracts.  However,  the  ability  of  the  program’s 
EVM  system  to  reliably  manage  and  forecast  the  program’s  cost  estimates 
may  be  affected  by  large  variances  between  the  systems  contractors’ 
estimated  cost  to  complete  operations  and  closure  versus  the 
government’s  estimate;  systems  contractors’  baselines  that  have  not  been 
carefully  controlled,  leading  to  high  percentages  of  authorized  work  that  is 
unpriced  for  long  periods  of  time;  and  limited  identification  of 
requirements  for  closure  costs  at  individual  sites. 

Because  of  the  shortcomings  in  the  underlying  cost  data,  the  usefulness  of 
the  EVM  data  to  make  cost  projections  is  limited  and  our  ability  to 
independently  verify  and  validate  the  reliability  of  the  cost  estimates  was 
restricted.  In  other  words,  the  mechanics  of  the  EVM  system  are  working 
properly;  however,  we  found  that  the  performance  management  baseline 
does  not  accurately  reflect  the  remaining  life  cycle  costs  of  the  Chemical 
Demilitarization  Program.  DOD’s  EVM  implementation  guide  states  that 
“the  “to-go”  plan  should  reflect  a  realistic  schedule  of  how  the  remaining 
work  actually  is  to  be  done  and  the  new  budget  should  be  adequate  and 
reflect  a  realistic  estimate,  remaining  program  risks,  and  contain  an 
appropriate  amount  of  management  reserve.”38  According  to  best 
practices,  an  EVM  system  should  be  devoid  of  data  errors  and  anomalies 
that  may  distort  the  analysis.39  In  addition,  any  cost  or  schedule  variances 
should  be  explained  and  corrective  actions  identified.  Furthermore,  the 
EVM  system  should  have  a  valid  and  stable  performance  baseline  from 


36Defense  Contract  Management  Agency,  Department  of  Defense  Earned  Value 
Management  Implementation  Guide  (Washington,  D.C.:  Apr.  7,  2005).  Also  see  DOD 
Memorandum,  Revision  to  DOD  Earned  Value  Management  Policy  (Washington,  D.C.: 
Mar.  7,  2005),  and  Office  of  Management  and  Budget,  Capital  Programming  Guide 
(Washington,  D.C.:  2006). 

3 ‘EVM  is  a  process  required  by  OMB  that  can  help  program  managers  track  program 
progress  by  using  the  value  of  work  done  as  a  basis  for  estimating  the  cost  to  compete  the 
project. 

38Defense  Contract  Management  Agency,  Department  of  Defense  Earned  Value 
Management  Implementation  Guide. 

39GAO-07-1 134SP. 
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which  to  make  projections  and  do  trend  analysis.40  In  the  case  of  the 
Chemical  Demilitarization  Program,  the  cost  account  managers  are 
inserting  their  costs  into  the  system  properly,  system  certification  verifies 
that  the  EVM  reported  costs  match  the  costs  in  the  accounting  system,  the 
EVM  reports  are  being  prepared  with  minimal  errors,  and  cost  account 
managers  are  explaining  any  variances.  However,  the  actual  data  are 
measured  against  a  baseline  that  does  not  reflect  all  of  the  work  to  be 
performed,  such  as  facility  closure  activities,  which  distorts  cost  and 
schedule  performance  metrics.  This  undermines  the  effectiveness  of  the 
EVM  system  in  providing  management  with  accurate  and  realistic  metrics 
for  making  informed  management  decisions.41 


Performance  Baselines  Our  review  showed  that  the  performance  baselines  lack  validity  and  have 

Lacked  Validity  and  Have  not  been  stable  because  CMA  and  site  systems  contractors  have  not  fully 

Been  LInstable  identified  all  requirements,  such  as  closure  costs,  and  included  them  in 

program  cost  estimates  and  because  large  amounts  of  additional  costs42 
are  added  to  the  performance  baseline  annually.  EVM  guidelines  state  that 
once  a  contract  is  awarded,  the  contractor  should  establish  a  performance 
baseline  that  will  serve  as  a  time-phased  budget  plan  for  accomplishing 
work  and  measuring  contract  performance.43  This  is  essentially  the 
spending  plan  for  the  program  and  should  closely  equate  to  the  planned 
budget,  which  is  the  value  of  work  planned  through  the  project’s 
completion.  The  performance  baseline  should  include  all  costs  associated 
with  completing  the  program,  including  direct  and  indirect  labor,  material, 
and  other  direct  costs  associated  with  the  authorized  work.  However,  we 
found  that  each  of  the  sites  was  lacking  a  stable  and  reliable  EVM 


40Defense  Contract  Management  Agency,  Department  of  Defense  Earned  Value 
Management  Implementation  Guide. 

41In  its  technical  comments  on  a  draft  of  this  report,  DOD  expressed  concern  about  our 
statement  that  the  EVM  system  used  at  each  site  may  not  be  reliable.  As  a  result,  the 
department  stated  that  it  will  request  the  Defense  Contract  Management  Agency  to  conduct 
an  audit  of  the  CMA  EVM  system  beginning  no  later  than  June  30,  2008. 

42The  systems  contractor  and  government  have  identified  work  that  (authorized  but 
unpriced)  is  negotiated  and  added  to  the  contract  price  annually. 

43Defense  Contract  Management  Agency,  Department  of  Defense  Earned  Value 
Management  Implementation  Guide.  Also,  see  GAO-07-1 134SP. 
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performance  baseline,  which  hindered  our  ability  to  perform  an 
independent  EVM  analysis.44 

While  the  planned  budget  and  the  performance  baseline  should  closely 
equate,  we  found  that  the  performance  baseline  is  significantly  less  than 
the  planned  budget  (close  to  $2  billion  less)  because  most  sites  have 
significant  amounts  of  work  that  are  authorized  by  the  systems 
contractors  and  the  Army  but  are  not  yet  placed  on  contract.  For  example, 
the  April  2007  SAR  reports  that  while  the  contract  value  at  Umatilla  is 
$1.2  billion  through  December  2006,  an  additional  $700  million  has  been 
authorized  but  is  pending  negotiation.  Also,  the  Estimate  at  Completion 
(EAC)  cost  estimate,  which  represents  all  estimated  costs  through  the  end 
of  the  contracts,  is  $4  billion  more  than  the  performance  baseline.  Figure  6 
shows  the  extent  that  planned  budget  and  the  EAC  exceed  the 
performance  baseline  for  all  sites  combined  over  a  19-month  period  from 
August  2006  to  February  2007. 


44In  its  technical  comments  on  a  draft  of  this  report,  DOD  expressed  concern  about  our 
statement  that  shortcomings  in  the  underlying  (EVM)  data  include  invalid  and  unstable 
performance  baselines.  DOD  stated  that  the  issues  with  the  contract  baselines  are  issues  of 
program  management  or  risk,  not  issues  of  EVM  data  or  process  reliability.  DOD  stated 
that  this  problem  has  been  ameliorated  by  the  recent  completion  of  life  cycle  contract 
negotiations  at  three  of  the  five  sites  covered  by  our  report  (Anniston,  Pine  Bluff,  and 
Umatilla).  The  contract  negotiations  for  Newport  and  Toole  are  ongoing  and  should  be 
completed  by  June  30,  2008.  The  department  stated  that  it  will  request  the  Defense 
Contract  Management  Agency  to  conduct  an  audit  of  the  CMA  EVM  system  beginning  no 
later  than  June  30,  2008. 
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Figure  6:  Comparison  of  the  Performance  Baseline  with  the  Planned  Budget  and  Systems  Contractors’  Estimate  at 
Completion 
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The  performance  baseline  does  not  include  all  requirements,  such  as 
closure  costs,  which  are  not  fully  identified  and  included  in  program  cost 
estimates.  Best  practices  state  that  a  cost  estimate  should  be 
comprehensive  in  order  to  be  considered  credible.45  Systems  contractor 
officials  stated  that  in  many  instances  what  was  included  in  their 
estimated  completion  costs  were  placeholder  amounts,  which  they 
believed  to  be  unrealistic.  According  to  program  office  officials,  the 
government  and  the  systems  contractors  have  not  yet  completely  defined 
the  requirements  for  closure,  so  a  realistic  closure  cost  estimate  is  not 
currently  available.  However,  since  two  sites — Johnston  Atoll  and 
Aberdeen — have  already  been  closed  historical  data  are  now  available  to 


45“Comprehensive”  means  that  the  cost  estimate  is  complete  and  accounts  for  all  possible 
costs.  Comprehensive  cost  estimates  completely  define  the  program,  reflect  the  current 
schedule,  and  are  technically  reasonable.  See  GAO-07-1 134SP. 


Page  43 


GAO-08-134  Chemical  Demilitarization 


develop  more  accurate  closure  estimates  at  the  remaining  sites.46  For 
example,  based  on  the  Army’s  experience  of  destroying  blister  and  nerve 
agents  at  Johnston  Atoll,  closure  costs  for  other  destruction  sites  are 
increasing  from  initial  estimates.  As  a  result  of  incorporating  these  actual 
data,  assumptions  for  conducting  closure  activities  have  increased  from  12 
months  to  about  30  months  at  some  sites. 

The  performance  baseline  also  is  unstable  because  large  amounts  of 
additional  costs  are  added  to  the  performance  baseline  annually.47  Each 
fiscal  year,  a  portion  of  the  unpriced  work  that  is  part  of  the  planned 
budget  is  negotiated  (priced),  and  it  is  moved  into  the  performance 
baseline.  The  negotiated  funding  is  often  greater  than  what  was  used  as  a 
“placeholder”  budget,  increasing  the  contract  baseline  to  more  than  what 
was  included  as  unpriced  work.  For  example,  the  fiscal  year  2006 
negotiated  increase  in  the  contract  value  at  Anniston  was  $6  million  more 
than  the  reduction  in  authorized  unpriced  work.  Program  office  officials 
explained  that  the  performance  baseline  at  each  site  is  constantly 
changing  because  of  yearly  contract  renegotiations48  for  future  operations 
and  closure  costs.  Program  officials  attribute  the  yearly  changes  to  the 
contracts  using  Operating  and  Maintenance  funds,  which  expire  within  1 
fiscal  year.  This  situation  created  the  need  to  put  on  contract  a  year’s 
worth  of  work  in  conjunction  with  each  fiscal  year’s  appropriation. 

CMA  officials  stated  that  they  are  making  efforts  to  stabilize  the  baseline 
by  placing  the  full  life  cycle  costs  associated  with  all  activities  at  each  site. 
In  its  technical  comments  on  a  draft  of  this  report,  DOD  stated  that  three 
of  the  five  CMA  sites  (Anniston,  Pine  Bluff,  and  Umatilla)  had  completed 
contract  performance  baselines  and  negotiations  were  ongoing  at  the  two 
remaining  sites  (Newport  and  Tooele).  DOD  expected  that  the 
negotiations  for  these  two  sites  would  be  completed  by  June  30,  2008. 

Once  this  happens,  CMA  officials  believe  that  there  will  be  less  of  a  need 
for  annual  negotiations  and  more  realistic  performance  baselines  should 
result.  Until  program  management  has  a  performance  baseline  that  reflects 
the  current  plan  to  complete  remaining  work  and  follows  a  disciplined 


46Johnston  Atoll  was  closed  in  December  2003  and  Aberdeen  was  closed  in  May  2007. 

4 ‘The  systems  contractor  and  government  have  identified  work  that  (authorized  but 
unpriced)  is  negotiated  and  added  to  the  contract  price  annually. 

48The  contracts  use  Operating  and  Maintenance  funds,  which  are  available  for  obligation 
for  1  fiscal  year.  This  creates  the  need  to  negotiate  the  contracts  on  a  yearly  basis,  since 
each  fiscal  year’s  funds  are  appropriated. 
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process  for  incorporating  changes,  management  will  not  be  able  to  benefit 
from  the  EVM  system’s  ability  to  reveal  actual  program  performance  and 
detect  program  risks. 


Program  Manager- 
Estimated  Completion 
Costs  Are  Significantly 
Higher  Than  Completion 
Costs  Generated  from 
EVM  Data,  Suggesting 
Data  Are  Incomplete 


It  is  a  best  practice  to  develop  a  range  of  independent  EACs  using 
commonly  accepted  EVM  equations  because  doing  so  informs 
management  of  the  likely  range  of  costs  for  completing  the  program  so 
appropriate  action  can  be  taken.49  In  addition,  DOD  guidance  states  that 
remaining  work  should  reflect  a  realistic  schedule  and  budget  that  also 
reflect  remaining  risks.50  Finally,  EVM  guidelines  state  that  program 
managers  should,  at  least  on  a  monthly  basis,  rely  on  EVM  data  to  generate 
EACs  that  are  derived  based  on  the  cost  of  work  completed  to  date  along 
with  an  estimate  of  what  it  will  cost  to  complete  the  rest  of  the  program.51 
By  relying  on  past  performance  and  using  well-defined  EVM  indices,  a 
variety  of  EACs  can  be  generated.  However  the  EACs  generated  are  only 
as  good  as  the  underlying  data  in  the  EVM  system. 


Based  on  our  analysis,  program  cost  estimates  often  exceeded  the  worst 
case  of  our  range  of  independently  developed  completion  costs  estimates 
because  of  shortcomings  in  the  underlying  data  of  the  EVM  systems 
maintained  at  the  chemical  demilitarization  sites.  It  is  a  best  practice  to 
develop  a  range  of  independent  cost  estimates  using  commonly  accepted 
EVM  equations.  Doing  so  informs  management  of  the  likely  range  of  costs 
for  completing  the  program  so  that  appropriate  action  can  be  taken.  We 
found  that  when  we  developed  independent  EACs  using  the  systems 
contractor’s  EVM  data  and  commonly  accepted  EVM  forecasting  equations 
in  all  cases  the  systems  contractors’  EACs  are  either  at  the  upper  end  of  or 
higher  than  our  estimated  EAC  range  based  on  our  analysis  shown  in  table 
10.  In  addition,  the  program  manager’s  EAC  was  far  outside  of  our 
projected  range  and,  in  many  cases,  was  almost  twice  as  high  as  our  EAC. 
For  example,  our  estimated  cost  range  for  Anniston  based  on  the  EVM 
data  was  from  $1.6  billion  to  $1.7  billion.  The  systems  contractor’s  most 
likely  cost  estimate  was  $100  million  above  the  upper  limit  of  our  range, 
and  the  program  manager’s  estimate  was  $1.4  billion  higher  than  the  upper 


49GAO-07-1 134SP. 

50Defense  Contract  Management  Agency,  Department  of  Defense  Earned  Value 
Management  Implementation  Guide. 

51GAO-07-1134SP. 
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range  limit.  We  attribute  this  difference  to  an  understated  performance 
baseline,  which  is  critical  to  the  estimating  process.  Table  10  compares 
our  independent  EAC  to  that  of  the  systems  contractor’s  and  program 
manager’s  EACs. 


Table  10:  Comparison  of  GAO’s  Independent  Estimates  to  Complete  Operations  and  Closure  Activities  with  the  Systems 
Contractor’s  and  Program  Manager’s  Estimates 


Then-year  dollars  in  billions 


Site 

Original  contract 
cost 

Current  planned 
budget 

Systems 
contractor’s  most 
likely  EAC 

GAO’s  independent 
EAC  range 

2006  Selected  Acquisition 
Report  program 
manager’s  EAC 

Anniston 

$0.6 

$1.7 

$1.8 

$1 .6-$1 .7 

$3.1 

Newport 

0.3 

1.0 

1.1 

0.9-1. 1 

1.8 

Pine  Bluff 

0.5 

1.5 

1.9 

1 .4-1 .5 

2.4 

Tooele 

0.2 

1.6 

2.4 

1 .4-1 .6 

3.3 

Umatilla 

0.5 

1.8 

1.9 

1.6-1 .8 

3.8 

Source:  GAO  analysis  of  Army  data. 


In  addition,  the  large  difference  between  the  program  manager’s  EAC  and 
our  independent  EACs  indicates  that  there  is  more  work  that  is  being 
considered  by  the  program  manager.  More  work,  in  turn,  will  affect  future 
costs  and  requirements  than  is  predicted  by  the  EVM  data  and,  therefore, 
calls  into  question  the  usefulness  of  the  EVM  data  to  effectively  manage 
the  program.52  The  ability  to  respond  quickly  to  program  problems 
depends  on  having  early  visibility  into  what  is  causing  them.  Thus,  if  the 
EVM  performance  baseline  is  missing  some  of  the  remaining  work,  then 
accurate  progress  assessments  will  not  be  available  to  provide 
management  with  a  better  picture  of  program  status.  This  lack  of  insight 
may  hinder  decision  making  and  program  success  because  problems  may 
go  undetected. 

Without  accurate  and  realistic  closure  requirements  used  to  develop 
completion  cost  estimates,  DOD  and  Congress  will  not  be  able  to 


52In  its  technical  comments  on  a  draft  of  this  report,  DOD  stated  that  the  Army  agrees  that 
there  is  usually  more  work  that  is  being  considered  and  not  yet  in  the  contract  baseline. 
The  Army  did  not  agree  that  this  situation  affects  the  EVM  data.  DOD  stated  that  the  Army 
uses  the  data  to  determine  current  variance  and  trend  information.  The  data  are  also  used 
by  the  systems  contractors  and  CMA  program  office  along  with  estimates  of  additional 
future  work  to  evaluate  the  life  cycle  schedule  and  costs.  The  department  stated  that  it  will 
request  the  Defense  Contract  Management  Agency  to  conduct  an  audit  of  the  CMA  EVM 
system  beginning  no  later  than  June  30,  2008. 
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accurately  identify  and  adequately  budget  for  all  activities  through  facility 
closure.  Similarly,  without  an  independent  cost  review  once  the  cost 
estimates  are  complete  and  schedule  milestones  have  been  updated  and 
reviewed,  DOD  and  Congress  will  not  have  assurance  that  cost  estimates 
for  the  closure  of  chemical  demilitarization  sites  are  realistic. 


The  Army  Has 
Actively  Managed 
Award  Fees  and 
Could  Further 
Emphasize  Schedule 
and  Cost  and  Link 
Award  Fees  to  More 
Objective 
Performance 
Measures 


Our  review  of  the  Army’s  use  of  monetary  fees  to  incentivize  certain 
aspects  of  the  systems  contractors’  performance  at  the  five  sites  in  or  near 
beginning  operations  in  2003  showed  that  the  contract  monetary  fee 
structures  at  the  incineration  sites  differed  from  those  at  Newport’s 
neutralization  site.  Award  fees  are  predominant  at  the  incineration  sites 
while  incentive  fees  are  predominant  at  Newport.  We  found  that  while  the 
Army  has  actively  reviewed  and  adjusted  the  types  of  monetary  fees  used 
and  the  criteria  and  weighting  for  determining  monetary  fees,  there  be  may 
be  opportunities  to  further  emphasize  schedule  and  cost.  For  example, 
during  the  period  we  reviewed,  award  fees  have  emphasized  safety  and 
environmental  compliance  at  each  site,  but  have  not  resulted  in  the 
control  of  schedule  and  cost  growth.  In  fiscal  year  2006,  after  several 
external  organizations  recommended  that  it  do  so,  the  Army  reinstituted 
the  use  of  a  distinct  award  fee  performance  measure  for  schedule.53 
However,  site  project  managers  often  weighted  schedule  at  the  lower  end 
of  the  specified  range.  Also,  while  DOD  has  published  guidance  that  award 
fees  should  be  tied  to  identifiable  interim  outcomes,  discrete  events,  or 
milestones  as  much  as  possible,  we  found  that  several  performance 
measures,  such  as  management  and  cost,  had  evaluation  factors  that  were 
all  or  mostly  all  subjective  and  were  not  clearly  linked  to  a  measurable 
outcome.  In  addition  to  award  fees,  in  2006,  the  Army  established  the 
Director’s  Programmatic  Performance  Based  Incentive  (DPPBI),  a 
performance-based  incentive  fee  to  encourage  the  incineration  site 
systems  contractors  to  establish  a  collaborative  relationship  with  the  goal 
to  share  lessons  learned  in  areas  that  would  result  in  schedule  reduction 
across  all  sites.  Before  fiscal  year  2007,  this  fee  was  based  on  subjective 
schedule  performance  measures.  By  fiscal  year  2007,  DPPBI  had  evolved 
to  focus  mostly  on  the  sites’  collective  schedule  performance  linked  to  the 
CWC  treaty  deadline.  It  is  unclear  how  effective  this  incentive  fee  will  be 
in  motivating  individual  systems  contractors’  schedule  performance  due  to 


53CMA  had  previously  incentivized  schedule  as  a  distinct  performance  measure  in  fiscal 
year  2002  at  all  four  incineration  sites  and  additionally  in  fiscal  years  2000  and  2001  at 
Tooele. 
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the  collective  manner  in  which  the  fee  is  earned,  its  relative  size,  and 
recent  focus  on  objective  schedule  performance.  An  additional  incentive 
fee,  authorized  by  section  923  of  the  National  Defense  Authorization  Act 
for  Fiscal  Year  2007,  provides  up  to  $165  million  at  each  incineration  site 
for  the  completion  of  destruction  and  facility  closure  by  or  close  to  the 
CWC  deadline.  This  incentive  fee,  as  planned,  is  structured  to  accelerate 
destruction  operations  and  facility  closure  at  the  incineration  sites. 
Because  this  fee  has  not  yet  been  implemented,  we  have  no  basis  at  this 
time  to  determine  how  effective  this  incentive  might  be  at  controlling 
schedule. 


Contract  Monetary  Fee 
Structures  at  the 
Incineration  Sites  Differ 
from  Those  at  Newport’s 
Neutralization  Site,  and 
Award  Fees  Are 
Predominant  at  the 
Incineration  Sites 


CMA  uses  cost-plus-award-fee  contracts  with  incentive  fees  to  encourage 
and  reward  the  systems  contractors  to  perform  in  an  innovative,  efficient, 
and  effective  way  in  areas  deemed  important  to  the  program’s  success. 
Under  these  contracts,  the  systems  contractors’  allowable,  incurred  costs 
are  reimbursable  to  the  extent  prescribed  in  the  contract.  According  to 
CMA  officials,  the  Chemical  Demilitarization  Program  systems  contracts 
are  cost  reimbursable,  based  on  the  fact  that  it  is  not  possible  to 
accurately  describe  the  requirements  in  sufficient  detail  to  address  the 
myriad  risks  and  uncertainties  associated  with  destroying  aging  and 
deteriorating  munitions  that  are  filled  with  chemical  agent.  As  a  result  of 
this  high-risk  environment,  the  government  accepts  most  of  the  cost  risk. 
Thus,  when  problems  arise,  such  as  rocket  fires  or  mercury 
contamination,  government  and  systems  contractor  resources  are  focused 
on  fixing  the  problems  or  mitigating  the  risk  rather  than  fixing  blame. 


The  base  fee  for  this  program  is  a  fixed  amount  of  money  negotiated  at  the 
beginning  of  each  year  of  contract  operations,  and  is  usually  paid  on  a 
monthly  basis  to  the  systems  contractor  for  its  performance,  though  this 
fee  is  not  tied  to  specific  performance  criteria  as  are  the  award  and 
incentive  fees.  According  to  program  officials,  the  rationale  for  providing  a 
base  fee  is  due  to  the  program’s  complexity  and  to  provide  the  systems 
contractors  with  a  level  cash  flow  for  financial  viability.  Award  fees  are 
intended  to  motivate  systems  contractor  performance  in  areas  that  are 
susceptible  to  judgmental  and  qualitative  measurement  and  evaluation 
and,  as  a  result,  award  fee  criteria  and  evaluations  tend  to  be  subjective. 
Typically,  award  fee  contracts  emphasize  multiple  aspects  of  contractor 
performance  in  a  wide  variety  of  areas,  such  as  quality,  timeliness, 
technical  ingenuity,  and  cost-effective  management.54  Finally,  incentive  fee 


54GAO-06-66. 
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contracts  use  what  is  considered  an  objective  evaluation  of  the  systems 
contractor’s  performance  to  adjust  the  fee  paid,  which  usually  involves 
applying  a  fee  determination  formula  specified  in  the  contract. 

The  contracts  at  each  of  the  four  operational  incineration  sites — Anniston, 
Pine  Bluff,  Tooele,  and  Umatilla — include  (1)  a  base  fee  (representing  3 
percent  of  the  negotiated  total  cost  of  the  contract),  which  is  a  fixed 
amount  not  subject  to  performance  evaluation;  (2)  an  award  fee  that  is  7 
percent  of  the  negotiated  contract  costs,  which  will  be  determined  through 
performance  evaluations  and  paid  from  an  award  fee  pool;  and  (3)  DPPBI, 
a  performance-based  incentive  fee  established  in  2005  and  paid  from  an 
available  fee  pool  amount  that  is  2  percent  of  negotiated  contract  cost.  The 
contract  at  the  remaining  neutralization  site — Newport — has  a  base  fee  (3 
percent),  an  award  fee  (3  percent),  and  a  performance-based  incentive  fee 
(6  percent).55  All  five  contracts  had  performance-based  incentive  fees 
added  after  initial  contract  award. 

The  types  and  percentages  of  fees  offered  to  the  systems  contractors  to 
safely  destroy  the  chemical  weapons  stockpile  differ  and  emphasize 
different  performance  measures.  For  example,  at  the  incineration  sites,  the 
award  fee  is  the  predominant  fee  and,  from  fiscal  year  2003  to  fiscal  year 
2007,  this  fee  has  largely  emphasized  safety  and  surety,  environmental 
compliance,  and  management  goals.  Conversely,  at  the  neutralization  site, 
the  performance-based  incentive  fee  is  the  predominant  fee,  which  has 
emphasized  schedule  goals.  Figure  7  compares  the  monetary  incentive 
fees  earned  by  the  systems  contractors  at  the  incineration  sites  and  at 
Newport.  As  shown  in  figure  7,  in  fiscal  year  2006,  CMA  paid  the  systems 
contractors  at  the  incineration  sites  a  combined  total  of  about 
$39.9  million  in  fees,  which  consisted  of  about  $31.1  million  (or 
78  percent)  in  award  fees  and  about  $8.8  million  (or  22  percent)  in 
performance-based  incentive  fees.  In  contrast,  CMA  paid  the  neutralization 
site  system  contractor  at  Newport  about  $14.7  million  in  monetary  fees, 
which  consisted  of  about  $2.5  million  (or  17  percent)  in  award  fees  and 
upwards  of  about  $12.1  million  (or  83  percent)  in  performance-based 


55The  initial  actual  percentages  of  the  base  and  award  fees  for  each  of  these  contracts  were 
originally  established  through  the  full  and  open  competitions  conducted  for  each  of  these 
contracts.  Each  contractor’s  proposal  included  a  percentage  for  both  the  base  fee  and  the 
award  fee,  which  the  government  evaluated  as  part  of  the  contract  selection  process.  With 
the  exception  of  one  systems  contractor,  all  the  contractors  proposed  a  total  fee  of  10 
percent  with  up  to  3  percent  proposed  for  the  base  fee  and  the  remaining  7  percent 
allocated  to  award  fee.  The  one  exception  is  Newport  where  the  contractor  proposed  a 
base  fee  of  3  percent  and  an  award  fee  of  3  percent. 
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incentive  fees.  In  fiscal  year  2006,  the  award  fees  earned  at  each  of  the 
incineration  sites  were  almost  three  times  higher  than  that  earned  at 
Newport. 


Figure  7:  Comparison  of  the  Monetary  Incentive  Fees  Earned  by  Systems  Contractors  at  the  Incineration  Sites  and  at  Newport 
during  Fiscal  Year  2006 


Incineration  sites’  monetary  fees  for 


Newport  monetary  fees  for  fiscal  year 
2006:  $14.7  million 


Award  fee 


Incentive  fee 


Source:  U.S.  Army  Sustainment  Command. 

Note:  GAO  did  not  independently  verify  the  award  and  incentive  fee  data  provided  by  the  U.S.  Army 
Sustainment  Command. 


By  predominately  using  performance-based  incentives  at  Newport,  the 
Army  has  aligned  more  of  its  monetary  fees  with  objective  outcomes. 
However,  Army  officials  said  that  it  is  difficult  to  determine  how  effective 
the  usage  of  performance-based  incentive  fees  has  been  in  motivating  the 
systems  contractors  to  meet  schedule  and  cost  milestones  because  of  the 
number  of  extensions  made  to  the  original  incentive  agreement.  The 
extensions  were  made  to  account  for  the  technical  challenges  encountered 
by  the  systems  contractors  during  the  neutralization  process.  After  the 
terrorist  attacks  of  September  11,  2001,  due  to  Army  concerns  about 
security  vulnerabilities,  CMA  restructured  the  contracts  to  increase  the 
emphasis  on  performance-based  incentives  at  the  Newport  and  Aberdeen 
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neutralization  sites  to  incentivize  the  systems  contractors  to  destroy  the 
remaining  stockpile  of  lethal  chemical  agent  on  an  accelerated  schedule.56 


Army  Actively  Managed 
Award  Fees  and  May  Be 
Able  to  Further  Emphasize 
Schedule  and  Cost 


Since  2003,  the  period  of  our  analysis,  the  Army  has  actively  reviewed  and 
adjusted  criteria  for  determining  monetary  fees — largely  award  fees — it 
provided  to  the  systems  contractors  to  incentivize  aspects  of  their 
performance.  CMA’s  award  fee  plans  generally  use  performance  measures 
that  could  be  expected  to  incentivize  the  desired  acquisition  outcomes. 
CMA  includes  broad  aspects  of  systems  contractor  performance,  such  as 
management  performance,  and  CMA  also  views  these  as  keys  to  a 
successful  program.  From  2003  through  2006,  CMA  used  safety  and  surety, 
environmental  compliance,  cost,  and  management  as  the  key  performance 
measures.  Until  fiscal  year  2006,  however,  CMA  did  not  emphasize 
schedule  as  a  distinct  performance  measure  in  the  award  fee  plans.57 


Army  Emphasized  Safety  and 
Environmental  Compliance  in 
Award  Fee  Plans  and  Did  Not 
Successfully  Incentivize 
Schedule  and  Cost 
Performance 


From  fiscal  year  2003  through  fiscal  year  2006,  the  incineration  sites  and 
Newport  had  not  been  able  to  maintain  original  production  schedule 
milestones  and  had  experienced  significant  cost  growth.  Although  the 
management  performance  measure  included  some  criteria  for  schedule, 
during  the  period  from  fiscal  year  2003  through  fiscal  year  2005,  schedule 
was  not  a  distinct  performance  measure  in  the  award  fee  plan,  and  safety 
and  environmental  compliance  were  heavily  weighted.  CMA  officials 
acknowledged  in  the  2007  incentive  agreement  plan  that  while  award  fees 
have  succeeded  in  motivating  the  systems  contractors  to  maintain  high 
levels  of  safety  and  environmental  compliance  at  each  site,58  they  have  not 
satisfactorily  resulted  in  the  control  of  schedule  and  cost  growth.  Many 
events  and  constraints  contributed  to  the  schedule  and  cost  growth,  both 
within  and  beyond  the  systems  contractors’  control.  CMA  officials  stated 
that  the  process  to  determine  the  degree  to  which  such  issues  cause 


56However,  these  incentives  did  not  keep  the  Newport  contract  on  schedule.  The  systems 
contractor  missed  the  target  completion  date  for  the  third  of  its  four  performance  incentive 
milestones  and  the  program  was  delayed  by  over  a  year.  According  to  DOD,  the  failure  to 
meet  this  milestone  was  due  to  unforeseen  technical  difficulties  and  could  not  have  been 
ultimately  influenced  by  any  type  of  contractual  language. 

57Schedule  was  a  distinct  performance  measure  in  the  Tooele  award  fee  plan  for  fiscal 
years  2000  and  2001.  In  2002,  it  was  a  distinct  performance  measure  in  the  award  fee  plans 
at  each  of  the  incineration  sites. 

58CMA  officials  noted  they  were  acting  in  accordance  with  Congress’s  mandate  that  the 
program  shall  provide  for  the  maximum  protection  of  the  workforce,  the  public,  and  the 
environment.  See  50  U.S.C.  1521  (c)(1)(A)  (2007). 
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variances  to  baselines  is  never  clear,  but  generally,  over  the  course  of  a 
year,  there  may  be  25  percent  of  the  problems  that  are  clearly  outside  of 
the  control  or  responsibility  of  the  systems  contractor,  10  percent  that  are 
clearly  within  the  control  of  the  systems  contractor,  and  65  percent  that 
fall  into  the  “gray  area”  where  it  is  difficult  to  determine  whether  the 
problem  is  clearly  outside  or  within  the  control  of  the  systems  contractor. 

From  fiscal  year  2003  through  fiscal  year  2006,  the  Army  paid  award  fees 
to  the  systems  contractors  between  70  percent  and  96  percent  of  the 
available  award  fee  pools  primarily  for  good-to-outstanding  performance 
in  safety  and  surety,  environmental  compliance,  and  management,  which 
collectively  constituted  most  of  the  basis  for  determining  the  award  fee. 
Table  11  shows  both  the  actual  amount  and  the  percentage  of  the  total 
available  award  fee  pool  each  site  was  paid  by  the  Army  over  the  period 
from  fiscal  year  2003 — when  all  five  sites  were  either  in  or  preparing  for 
the  operations  phase — to  fiscal  year  2006.  Based  on  our  analysis  of  this 
period,  the  Army  paid  the  systems  contractors  at  three  of  the  four 
incineration  sites  award  fees  between  86  percent  and  96  percent  of  the 
total  award  fee  pool  available  during  this  period.  The  systems  contractor 
at  Tooele  was  the  exception,  earning  only  70  percent  of  the  total  available 
award  fee  pool,  according  to  the  U.S.  Army  Sustainment  Command, 
because  of  unsatisfactory  ratings  due  to  the  lack  of  real  progress  in 
planning  and  preparing  for  the  mustard  campaign  and  in  not  meeting 
government  expectations  for  managing  costs. 


Table  1 1 :  Percentage  of  Award  Fees  Earned  at  the  Incineration  Sites  and  Newport,  from  Fiscal  Year  2003  to  Fiscal  Year  2006 

Chemical 

demilitarization 

facility 

Amount  of  award  fee  pool 
(2003-2006) 

Amount  of  award  fee  earned 
(2003-2006) 

Percentage  of  award  fee  earned 
(2003-2006) 

Anniston 

$24,536,587 

$23,671,150 

96 

Pine  Bluff 

22,564,501 

19,633,254 

87 

Umatilla 

26,235,256 

22,638,970 

86 

Newport 

10,532,717 

8,734,713 

83 

Tooele 

27,917,118 

19,414,301 

70 

Total 

$111,786,179 

$94,092,388 

Source:  U.S.  Army  Sustainment  Command. 


Anniston,  for  example,  earned  $23.7  million  (or  96  percent)  of  the  award 
fee  during  this  period.  According  to  an  award  fee  evaluation  board  report, 
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while  the  Anniston  systems  contractor  worked  aggressively  to  implement 
the  schedule  without  compromising  worker  or  public  safety,  the  systems 
contractor  faced  challenges  related  to  disciplined  operations.59  The  report 
noted  that  certain  destruction  campaigns  at  Anniston  were  characterized 
by  schedule  delays  that  resulted  primarily  from  equipment  reliability 
problems  and,  at  times,  inefficient  planning  and  execution  of  repair 
efforts.  The  report  further  noted  that  while  the  efficiency  of  repairs  was  at 
times  problematic,  the  systems  contractor  could  not  have  prevented  the 
equipment  failure.  The  report  also  credited  the  systems  contractor’s  risk 
mitigation  efforts  with  significantly  reducing  the  risk  of  a  6-month 
potential  site  shutdown  that  would  have  carried  with  it  a  corresponding 
cost  impact  of  $75  million.  Thus,  CMA  was  able  to  justify  paying  the 
systems  contractor  nearly  the  full  amount  of  available  award  fee.  Also, 
during  one  period,  Umatilla’s  systems  contractor  earned  $2.5  million  (or  80 
percent)  of  the  available  award  fee  pool  while  it  had  destroyed  36  percent 
of  the  nerve  agent  rockets  planned  for  that  period.  According  to  an  award 
fee  evaluation  board  report,  the  processing  shortfall  was  attributable  to 
the  systems  contractor’s  lack  of  implementation  of  planned  and  emergent 
work  in  an  efficient  and  effective  manner,  and  was  estimated  to  increase 
the  planned  operations  schedule  by  77  days.60  We  have  no  basis  for  saying 
that  the  systems  contractors  should  not  have  received  their  award  fees, 
since  they  generally  were  evaluated  as  having  met  the  award  fee  criteria, 
which  at  the  time  did  not  have  schedule  as  a  distinct  performance 
measure. 

CMA  officials  have  acknowledged  that  the  existing  award  fee  process  has 
not  satisfactorily  resulted  in  the  control  of  schedule  and  cost  growth.  CMA 
developed  the  schedule  performance  measure  in  response  to  Army 
Inspector  General  recommendations,  and  DOD  guidance  suggested  that 
CMA  focus  more  attention  on  schedule  criteria.  The  Army’s  reinstatement 
of  a  distinct  schedule  performance  measure  coincided  with  the  program’s 
schedule  extensions.61  Also,  in  fiscal  year  2004,  according  to  CMA,  the 


59See  the  Award  Fee  Review  Board  Report  for  the  Anniston  Chemical  Agent  Disposal 
Facility,  Evaluation  Number  17,  for  the  evaluation  period  from  April  1,  2005,  to  September 
30,  2005. 

60See  the  Award  Fee  Evaluation  Board  Report  for  the  Umatilla  Chemical  Agent  Disposal 
Facility,  First  Period,  Fiscal  Year  2005,  for  the  period  of  performance  from  October  1, 
2004,  to  March  31,  2005. 

“Memorandum  from  the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and 
Logistics  on  Chemical  Demilitarization  Program  Strategic  Governance  (Washington,  D.C.: 
Jan.  6,  2006). 
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Army  Has  Adjusted  Award  Fee 
Plans’  Weighted  Performance 
Measures  and  Could  Further 
Increase  the  Weight  of 
Performance  Measures,  Such 
as  Schedule  and  Cost 


Army  Inspector  General  recommended  that  CMA  develop  quantifiable 
metrics  and  both  the  Army  Inspector  General  and  the  Defense  Contract 
Management  Agency  recommended  that  CMA  place  greater  weight  on  cost 
and  schedule  control.  CMA  documents  indicate  that  the  schedule 
performance  measure  was  developed  after  these  recommendations  and 
the  Under  Secretary’s  memorandum. 

The  determination  of  award  fee  performance  measures  and  related  weight 
factors  is  part  of  CMA’s  award  fee  plan  template  development  process, 
which  is  used  as  a  guideline  by  site  project  managers  to  develop  specific 
site  award  fee  plans.  The  template  describes  specific  criteria,  standards, 
and  procedures,  and  provides  a  range  of  weight  for  each  performance 
measure  to  be  used  in  the  award  fee  plan  to  assess  the  systems 
contractors’  performance  and  to  determine  the  amount  of  award  fee 
earned.  The  template  allows  the  site  project  managers  flexibility  to  select  a 
weighted  factor  for  each  performance  measure  within  the  CMA  template’s 
range  based  upon  the  site  project  manager’s  discretion.  For  example,  the 
award  fee  plan’s  fiscal  year  2007  template  had  a  range  of  10  percent  to  20 
percent  for  weighting  management  performance,  20  percent  to  30  percent 
for  safety  and  surety,  20  percent  to  25  percent  for  environmental 
compliance,  10  percent  to  15  percent  for  cost  performance,  and  10  percent 
to  20  percent  for  schedule  performance.  The  Army  adjusted  these  ranges 
in  fiscal  year  2006,  adding  the  schedule  performance  measure,  increasing 
the  template’s  range  for  management,  and  reducing  the  template’s  ranges 
for  environmental  compliance  and  safety  and  surety  performance 
measures. 

Table  12  shows  historical  data  on  the  award  fee  plans’  performance 
measures  and  the  relative  weights  selected  by  each  of  the  incineration 
sites  and  Newport  as  well  as  the  adjustments  made  from  fiscal  year  2003 
to  fiscal  year  2007. 
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Table  12:  Performance  Measures  and  Weights  in  Award  Fee  Plans,  Fiscal  Years  2003-2007 


Percentages 


Performance  measures 

2003 

2004 

2005 

2006 

2007 

Management 

Pine  Bluff 

N/A 

15 

15 

20 

20 

Umatilla 

N/A 

15 

15 

20 

20 

Anniston 

N/A 

15 

15 

20 

20 

Tooele 

15 

15 

15 

20 

15 

Newport 

25 

25 

15 

15 

20 

Safety  and  surety 

Pine  Bluff 

35 

40 

40 

30 

30 

Umatilla 

35 

40 

40 

30 

30 

Anniston 

35 

40 

40 

30 

30 

Tooele 

40 

40 

40 

30 

35a 

Newport 

30 

30 

35 

25 

30 

Environmental  compliance 

Pine  Bluff 

30 

35 

35 

25 

25 

Umatilla 

30 

35 

35 

25 

25 

Anniston 

30 

35 

35 

25 

25 

Tooele 

35 

35 

35 

25 

25 

Newport 

30 

30 

35 

25 

20 

Schedule 

Pine  Bluff 

N/A 

N/A 

N/A 

15 

15 

Umatilla 

N/A 

N/A 

N/A 

15 

15 

Anniston 

N/A 

N/A 

N/A 

10 

10 

Tooele 

N/A 

N/A 

N/A 

15 

15 

Newport 

N/A 

N/A 

N/A 

10 

15 

Costs 

Pine  Bluff 

15 

10 

10 

10 

10 

Umatilla 

15 

10 

10 

10 

10 

Anniston 

15 

10 

10 

15 

15 

Tooele 

10 

10 

10 

10 

10 

Newport 

15 

15 

15 

20 

15 
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Percentages 

Performance  measures 

2003 

2004 

2005 

2006 

2007 

Technical  performance 

Pine  Bluff 

20 

N/A 

N/A 

N/A 

N/A 

Umatilla 

20 

N/A 

N/A 

N/A 

N/A 

Anniston 

20 

N/A 

N/A 

N/A 

N/A 

Tooele 

N/A 

N/A 

N/A 

N/A 

N/A 

Newport 

N/A 

N/A 

N/A 

N/A 

N/A 

Source:  U.S.  Army  Sustainment  Command. 


Notes:  Data  on  the  award  fee  plans  included  fiscal  year  2003  through  fiscal  year  2007  to  represent 
the  period  of  time  the  sites  were  in  or  preparing  for  the  operations  phase.  In  addition,  we  evaluated 
the  performance  measures  and  related  weights  for  those  fiscal  years  since  the  previous  APB  was 
established  in  April  2003.  N/A  indicates  that  no  weight  was  assigned  to  the  particular  award  fee  plan 
performance  measure  at  that  site  for  that  particular  fiscal  year. 

“Site  project  managers  can  select  weights  outside  of  suggested  ranges  with  CMA  approval.  While  the 
award  fee  plan’s  fiscal  year  2007  template  had  a  range  of  20  percent  to  30  percent  for  weighting 
safety  and  surety,  Tooele’s  weight  for  safety  and  surety  for  fiscal  year  2007  was  35  percent. 

As  shown  in  table  12,  government  site  project  managers  have  weighted 
safety  and  surety  and  environmental  compliance  from  a  weight  factor  that 
ranged  from  30  percent  to  40  percent  in  fiscal  year  2003  to  a  weight  factor 
from  20  percent  to  35  percent  in  fiscal  year  2007.  Also,  the  award  fee  plans 
for  the  incineration  sites  and  Newport  have  safety  “killer  clause” 
provisions,  whereby  a  systems  contractor  could  lose  all  or  part  of  its 
award  fee  if  there  is  an  agent  release,  an  individual  or  worker  is  exposed 
to  chemical  agent,  or  there  is  a  release  of  agent  outside  of  the  engineering 
controls. 

In  fiscal  year  2006,  CMA  added  the  schedule  performance  measure, 
increased  the  template’s  range  for  management,  and  reduced  the 
template’s  range  for  environmental  compliance  and  safety  and  surety.  We 
found,  based  on  our  analysis,  that  the  site  project  managers  consistently 
placed  schedule  and  cost  performance  at  lower  relative  weights  than  other 
factors,  including  management  performance.  For  example,  in  fiscal  years 
2006  and  2007,  the  maximum  weight  allowable  by  CMA  for  schedule 
performance  was  20  percent,  but  the  average  weight  assigned  by  the  site 
project  managers  in  the  award  fee  plans  was  only  13.75  percent  for  the 
incineration  sites.  Conversely,  the  site  project  managers  weighted  other 
performance  measures,  such  as  management,  on  average  much  closer  to 
CMA’s  maximum  suggested  weights. 

Program  officials  stated  that  the  weights  assigned  to  the  different 
performance  measures  are  based  on  the  relative  significance  of  the  criteria 


Page  56 


GAO-08-134  Chemical  Demilitarization 


to  the  program.  Due  to  the  lethal  nature  of  the  material  that  must  be 
processed,  Congress  has  mandated  that  the  program  ensure  maximum 
protection  to  the  workforce,  the  public,  and  the  environment.  Accordingly, 
as  program  officials  point  out,  safety/surety  carry  the  highest  weighting 
followed  closely  by  environmental  compliance.  However,  given  the 
maturity  of  the  operations  phase,  there  may  be  opportunities,  from  a 
management  perspective,  for  the  Army  to  further  adjust  weights  for 
performance  measures  in  the  award  fee  plan  template  to  further 
incentivize  key  acquisition  outcomes,  such  as  schedule  and  cost,  at  the 
incineration  sites.  The  five  incineration  sites  have  been  preparing  for  or 
were  in  operation  since  at  least  2003  and,  along  with  Newport  and 
Aberdeen,  have  eliminated  over  45  percent  of  the  chemical  munitions 
stockpile.  In  addition,  the  Army  has  stated  that  it  has  reduced  much  of  its 
safety  risk  by  destroying  the  bulk  of  its  most  lethal  and  volatile  chemical 
munitions.  For  example,  the  Tooele  site  has  mostly  ton  containers  with 
mustard  agent  left  to  destroy,  which  is  considered  less  risky  than  other 
agents.  Moreover,  Army  officials  have  pointed  out  that  the  systems 
contractors  have  implemented  safety  improvement  plans  that  have 
incorporated  appropriate  protocols  and  standards  into  their  daily 
operations.  As  a  result,  from  a  management  perspective,  there  may  be 
opportunities  to  further  emphasize  schedule  and  cost  without  sacrificing 
safety  and  environmental  compliance. 


It  Is  Possible  to  Make 
Greater  Use  of  Objective 
Award  Fee  Criteria  and 
Evaluation  Factors 


Based  on  our  review  of  the  fiscal  year  2007  award  fee  plan’s  performance 
measures  and  related  evaluation  criteria  for  the  sites,  most  of  the  award 
fee  criteria  and  evaluation  factors  are  not  objective  and  measurable.  We 
determined  that  it  is  possible  to  make  CMA’s  award  fee  plans  more 
effective  in  achieving  certain  acquisition  outcomes  through  greater  use  of 
objective  award  fee  criteria.  We  previously  recommended  that  DOD 
instruct  the  military  services  to  move  toward  more  outcome-based  award 
fee  criteria  that  are  achievable  and  promote  accountability  for  and  are 
directly  linked  to  achieving  desired  program  outcomes.62  In  March  2006, 
DOD  issued  award  fee  guidance  stating  that  it  is  imperative  that  award 
fees  be  tied  to  identifiable  interim  outcomes,  discrete  events,  or 


62GAO-06-66  and  GAO,  NASA  Procurement:  Use  of  Award  Fees  for  Achieving  Program 
Outcomes  Should  Be  Improved,  GAO-07-58  (Washington,  D.C.:  Jan.  17,  2007). 
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milestones  as  much  as  possible.63  We  acknowledge  that  award  fee 
contracts  are  expected  to  use  subjective  criteria  to  the  extent  that 
objective  criteria  are  not  feasible.  However,  we  identified  additional 
opportunities,  from  a  management  perspective,  for  CMA  to  make  greater 
use  of  objective  award  fee  criteria  and  evaluation  factors  to  incentivize 
contractors  to  achieve  desired  acquisition  outcomes,  such  as  safely 
destroying  the  chemical  stockpile  on  schedule  and  within  cost. 

The  award  fee  plans  for  this  program  make  use  of  both  subjective  and 
objective  criteria  and  our  analysis  of  the  fiscal  year  2007  award  fee  plan 
showed  that  in  most  cases,  the  criteria  are  subjective.  Three  performance 
measures — safety,  environmental  compliance,  and  schedule — use  a  mix  of 
objective  and  subjective  performance  evaluation  criteria,  although  we 
noted  these  evaluations  can  also  factor  in  subjective  assessments.  The 
award  fee  plan  states  that  in  evaluating  performance  in  these  areas,  a 
number  of  subjective  attributes  will  be  considered,  but  there  are  no 
specific  weights  associated  with  the  attributes.  Rather,  they  are 
considered  guides  to  evaluate  the  broad  spectrum  of  potential  systems 
contractor  performance.  The  schedule  performance  measure  was,  for  the 
most  part,  objective  and  linked  to  an  outcome — the  destruction  of 
chemical  munitions.  For  example,  the  2007  schedule  performance 
category  used  site-specific  destruction  tonnage  goals  related  to  meeting 
CWC  deadlines  and  the  APB  schedule.  On  the  other  hand,  three  other 
performance  measures — cost,  management,  and  surety — are  evaluated 
using  all  subjective  criteria.  While  we  recognize  that  using  subjective 
criteria  to  assess  systems  contractor  performance  is  acceptable,  we 
identified  opportunities  for  the  Army  to  use  more  objective  award  fee 
criteria  when  evaluating  contractor  performance. 


63Memorandum  from  the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and 
Logistics  on  Award  Fee  Contracts  (FAR  16,  DFARS  21S,  DFARS  216)  (Washington,  D.C.: 
Mar.  29,  2006).  In  addition,  the  National  Defense  Authorization  Act  for  Fiscal  Year  2007 
directed  DOD  to  issue  guidance,  with  detailed  implementation  instructions,  to  ensure  that 
all  new  contracts  using  award  fees  link  such  fees  to  acquisition  outcomes,  which  are 
defined  in  terms  of  program  cost,  schedule,  and  performance.  See  Pub.  L.  No.  109-364, 

§  814  (2006).  Furthermore,  in  an  April  2007  memorandum,  DOD  issued  further  award  fee 
guidance  stating  that  it  is  the  policy  of  the  department  that  for  new  solicitations  issued  on 
or  after  August  1,  2007,  objective  criteria  be  utilized,  whenever  possible,  to  measure 
contract  performance.  If  it  is  determined  that  objective  criteria  do  not  exist,  then  the  head 
of  the  contracting  activity  must  sign  a  determination  and  finding  that  “the  work  to  be 
performed  is  such  that  it  is  neither  feasible  nor  effective  to  devise  predetermined  objective 
incentive  targets  applicable  to  cost,  technical  performance,  or  schedule.” 
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We  developed  some  illustrative  examples  to  show  how  certain  elements  of 
CMA’s  subjective  criteria  taken  from  the  2007  award  fee  template  could  be 
revised  to  objectively  measure  systems  contractor  performance.  We 
recognize  that  there  are  other  examples  that  could  be  used  to  illustrate 
this  point  and  do  not  suggest  that  these  examples  necessarily  be  used.  To 
develop  these  examples,  we  used  as  criteria  our  analysis  and  prior  work.64 
For  example,  the  current  award  fee  plan  contains  the  following  subjective 
criterion  for  evaluating  management  performance:  “Demonstrated  ability 
to  practice  proactive  management  to  identify  and  anticipate  problems  and 
implement  effective  countermeasures  prior  to  adverse  impact.”  Illustrative 
examples  of  potential  objective  criteria  that  could  address  these  criteria 
are  “Develop  and  implement  an  integrated  risk  management  plan  by  [a 
specified  date]”  and  “Ensure  that  [a  specified  percentage]  of  the  risks 
identified  have  a  countermeasure  available  and  implemented  within  [a 
specified  number  of  days]  of  the  mustard  container  campaign.” 

Similarly,  the  current  award  fee  plan  contains  the  following  subjective 
criterion  for  evaluating  cost  performance:  “Cost  management  systems 
(accounting,  scheduling,  planning,  and  budgeting)  are  all  efficiently  and 
effectively  integrated,  and  produce  consistent,  reliable,  and  accurate  data.” 
An  illustrative  example  of  a  potential  objective  criterion  that  could 
address  at  least  a  portion  of  these  criteria  is  “Ensure  that  [a  specified 
percentage]  of  the  cost  and  schedule  management  systems  data  accurately 
depicts  contract  cost  status  within  the  assessment  period.”  Moreover,  the 
current  award  fee  plan  contains  the  following  subjective  criterion  for 
evaluating  surety  performance:  “Appropriate  treatment  and  disposal  of 
chemical  agents  and  chemical  agent  standards.”  An  illustrative  example  of 
a  potential  objective  criterion  that  could  address  at  least  a  portion  of  these 
criteria  is  “Ensure  that  [a  specified  percentage]  of  chemical  agents  are 
appropriately  treated  and  disposed  of  under  the  chemical  agent 
standards.” 

After  the  March  2006  guidance  on  award  fees  was  issued  by  the  Under 
Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics,  CMA 
issued  a  policy  on  award  fees  in  September  2006.  CMA’s  policy  stated  that 
“due  to  the  complexity  of  the  program  and  the  toxic  nature  of  the  material 
being  processed,  essential  elements  of  performance  cannot  effectively  or 
feasibly  be  measured  by  predetermined  objective  metrics.  Therefore, 
award  fee  criteria  are  established  that  allow  assessment  of  performance 


64See  GAO-06-66  and  GAO-07-58. 
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through  the  use  of  objective  and/or  subjective  metrics.”  Our  review  of  this 
policy  indicated  that  CMA  did  not  clearly  or  fully  incorporate  the  concept 
of  tying  “award  fees  to  identifiable  interim  outcomes,  discrete  events  or 
milestones  as  much  as  possible”  as  called  for  in  the  March  2006  guidance. 
Although  not  legally  required,  we  believe  that  tying  award  fees  as  much  as 
possible  to  more  objective,  measurable  evaluation  criteria  is  a  sound 
business  and  management  practice,  and  that  CMA  could  do  more  to  fully 
incorporate  and  implement  the  March  2006  DOD  policy  guidance. 


Recently  Established  and 
Planned  Performance- 
Based  Incentive  Fees  Are 
Focused  on  Schedule,  but 
Their  Effectiveness  Is 
Uncertain  at  This  Time 


The  Army  has  recently  offered  monetary  fees  intended  to  provide 
additional  incentives  to  improve  schedule  performance.  In  2006,  the  Army 
added  to  its  fee  structure  the  DPPBI,  which  is  an  annual  performance- 
based  incentive  fee  that  is  structured  to  encourage  the  systems 
contractors  at  the  four  incineration  sites  to  work  collaboratively  to 
improve  various  aspects  of  their  performance.  That  is,  while  each 
individual  site  is  evaluated  and  scored  separately,  the  individual  scores  are 
averaged  to  determine  a  composite  score  that  results  in  all  sites  earning 
the  same  percentage  of  the  available  DPPBI  fee  pool.  Since  the  initial 
evaluation  period  in  2006,  the  fee  amount  available  for  the  DPPBI  has  been 
2  percent  of  the  estimated  fee-bearing  contract  costs — relatively  small 
when  compared  to  the  award  fee  available  to  the  incineration  sites.  The 
2  percent  DPPBI  is  in  addition  to  the  10  percent  available  through  base 
and  award  fees. 


The  evaluation  factors  for  assessing  systems  contractor  performance  for 
the  DPPBI  have  changed  over  time  and  are  divided  between  objective 
criteria  and  a  subjective  assessment  of  systems  contractor  performance 
during  the  fiscal  year.  Three  criteria  were  established  for  the  initial  DPPBI 
evaluation  period  of  January  1  through  September  30,  2006:  risk 
mitigation,  production  assessment,  and  lessons  learned.  In  the  initial 
evaluation  period,  the  contractors  collectively  were  deemed  to  have 
earned  61  percent  of  the  2006  DPPBI  pool  or  $3,313,964  out  of  $6,497,949. 
In  fiscal  year  2006,  the  DPPBI  criteria  were  changed  to  put  more  emphasis 
on  performance  outputs  and  less  on  processes  and  procedures.  The  fiscal 
year  2006  DPPBI  criteria  were  established  to  encourage  the  contractors  to 
collectively  reduce  project  schedules  while  maintaining  a  high  standard  of 
surety,  safety,  and  environmental  compliance.  The  criteria  also  were 
established  to  increase  CMA’s  confidence  levels  that  the  sites  could 
achieve  CWC  destruction  deadlines,  although  discrete  goals  were  not 
specified.  For  the  fiscal  year  2006  period  of  performance,  the  contractors 
earned  about  93  percent  or  $8,771,200  out  of  the  $9,411,159  in  the  DPPBI 
pool  for  that  period. 
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Prior  to  fiscal  year  2007,  the  DPPBI  was  not  linked  to  objective  schedule 
performance  criteria  linked  to  the  April  29,  2012,  CWC  treaty  deadline. 
Beginning  in  fiscal  year  2007,  the  DPPBI  fee  evaluation  was  linked  to 
achieving  discrete  schedule  milestones  associated  with  achieving  the  CWC 
April  29,  2012,  deadline.  For  fiscal  year  2007,  70  percent  of  the  total  DPPBI 
fee  available  to  be  earned  (or  up  to  a  maximum  of  1.4  percent  of  the  total 
negotiated  contract  costs,  depending  on  average  collective  performance) 
was  based  on  specified  schedule  attainment  targets  for  each  site.  The 
remaining  30  percent  of  the  available  DPPBI  fee  for  fiscal  year  2007  is 
based  on  subjective  criteria  related  to  the  contractors’  performance  in  the 
application  of  lessons  learned,  risk  management,  programmatic 
procurement,  and  other  collaborative  approaches  to  reduce  costs  and 
accelerate  destruction  of  the  stockpile.  This  fee  would  be  in  addition  to 
the  portion  of  the  award  fee  targeted  at  schedule  performance,  now  about 
1  percent  of  total  negotiated  contract  costs  at  most  incineration  sites.  This 
additional  emphasis  on  schedule  is  a  positive  development,  but  it  remains 
unclear  to  us  how  effective  this  incentive  fee  will  be  in  motivating 
individual  systems  contractors’  schedule  performance  due  to  the  collective 
manner  in  which  the  fee  is  earned,  its  relative  size,  and  recent  focus  on 
objective  schedule  performance.  It  is  particularly  unclear  to  us  how 
individual  contractors  might  be  motivated  during  periods  when  the 
underperformance  of  one  or  more  systems  contractors  could  greatly 
reduce  the  potential  to  earn  the  fees  by  the  others. 

Finally,  the  Army  plans  to  use  a  recently  authorized  incentive  fee  of  up  to 
$165  million  per  site  in  an  effort  to  further  motivate  the  systems 
contractors  to  accelerate  destruction  and  closure  activities.  Although  it 
had  not  been  placed  on  contract  at  the  time  of  our  review,  CMA  drafted  a 
plan  to  offer  incentive  fees  under  the  authority  contained  in  section  923  of 
the  National  Defense  Authorization  Act  for  Fiscal  Year  2007. 65,66  This  plan 
would  allow  systems  contractors  at  the  four  incineration  sites  the 
opportunity  to  earn  fees  in  addition  to  the  12  percent  available  to  be 
earned  annually.  As  currently  planned,  systems  contractors  will  be  able  to 
earn  incentive  payments  for  completion  of  destruction  operations  tied  to 
dates  around  the  final  CWC  deadline  and  for  the  acceleration  of  facility 
closure  activities,  but  the  effectiveness  of  this  approach  is  uncertain. 


65Pub.  L.  No.  109-364  (2006). 

“Chemical  Materials  Agency,  Incentive  Agreement  for  Completion  of  Destruction 
Operations  and  Facility  Closure  Activities  at  Chemical  Stockpile  Disposal  Facilities 
(Washington,  D.C.:  May  19,  2007). 
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Under  CMA’s  plan,  it  would  offer  the  additional  incentive  fee  of  up  to 
$150  million  to  each  site — up  to  $100  million  for  destruction  operations 
and  up  to  $50  million  for  facility  closure  activities  completed  within 
specified  target  ranges.  Bonuses  totaling  $15  million  would  also  be 
available  to  each  site  if  all  sites  complete  destruction  operations  before 
the  treaty  deadline  and  accelerate  facility  closure  activities. 

CMA’s  plan,  which  would  have  to  be  implemented  under  each  of  the 
contracts,  has  a  number  of  positive  features.  For  example,  the  plan 
recognizes,  as  does  the  statute,  that  the  enterprise  of  striving  to  capture 
these  incentives  is  to  be  conducted  consistent  with  the  existing  obligation 
to  provide  for  maximum  protection  for  the  environment,  the  general 
public,  and  the  personnel  who  are  involved  in  the  destruction  of  the 
chemical  agents  and  munitions.  The  potential  effectiveness  of  this 
particular  planned  incentive  at  motivating  the  systems  contractors  to 
complete  destruction  operations  around  the  treaty  deadline  is  also 
uncertain.  This  incentive  arrangement  has  not  yet  been  placed  on  contract 
and  thus  it  is  too  early  to  assess  its  potential  success.  Additional  details 
about  the  two  recently  established  incentive  fees  are  in  appendix  II. 


Conclusions 


Destroying  the  nation’s  remaining  stockpile  of  chemical  weapons  in  a  safe, 
efficient,  and  timely  manner  is  essential  to  improve  the  potential  for  the 
United  States  to  meet  CWC  obligations  and  to  reduce  the  risk  to  the  public 
and  the  environment  of  a  potential  catastrophic  event  associated  with 
lethal  chemical  stockpiles.  We  recognize  that  maintaining  high  levels  of 
safety  and  environmental  compliance  are  inextricably  linked  to  success  in 
meeting  schedule  and  cost  goals.  We  also  recognize  that  it  is  highly 
important  that  program  management,  schedule  and  cost  estimates,  and 
monetary  incentives  be  soundly  structured,  well  aligned  with  program 
goals  and  desired  acquisition  outcomes,  reasonable,  and  reliable.  While 
DOD  has  taken  specific  actions  to  strengthen  program  management, 
improving  the  focus  of  annual  performance  plans  beyond  a  single  year  and 
refining  and  finalizing  planned  actions  in  the  risk  management  area  could 
greatly  enhance  the  usefulness  of  these  management  tools  for  DOD,  CMA, 
the  sites,  and  Congress.  In  addition,  without  a  fully  developed  and 
implemented  risk  management  approach,  CMA  cannot  ensure  that  it  is 
fully  anticipating  and  mitigating  risks  going  forward.  Without  such 
improvements,  it  would  be  difficult  for  program  officials  to  accurately 
track  progress,  effectively  initiate  timely  actions  to  mitigate  risks,  and 
increase  the  potential  for  program  activities  to  further  contribute  to  the 
overall  program  goal  of  safely  destroying  the  chemical  weapons  stockpile 
in  a  safe  and  environmentally  compliant  manner. 
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Similarly,  it  is  important  that  DOD,  CMA,  and  Congress  have  reasonable 
schedule  and  cost  estimates  with  reliable  supporting  data  and  other 
assumptions,  including  when  they  change  significantly,  either  increasing 
or  decreasing  funding  requirements.  With  most  of  the  chemical  agent 
destruction  sites  significantly  ahead  of  the  2005  program  office  schedule, 
the  program  milestones  appear  to  be  overly  conservative,  regardless  of 
unexpected  risks,  such  as  the  rocket  fires  at  Umatilla.  As  a  result,  the 
program  schedule  milestones  may  no  longer  be  a  realistic  baseline  for 
measuring  program  performance  and  warrant  further  review  and 
independent  assessment  to  give  DOD  and  Congress  greater  confidence 
that  the  program  is  being  held  accountable  to  a  reasonable  baseline  to 
measure  the  program’s  progress.  In  addition,  the  Chemical 
Demilitarization  Program’s  revised  APBs  and  its  most  recent  SARs — key 
management  and  oversight  documents — lack  up-to-date  and  reviewed 
schedule  milestone  information.  Furthermore,  DOD  needs  accurate  and 
realistic  closure  cost  estimates  and  other  cost  inputs  for  each  of  DOD’s 
chemical  demilitarization  sites  to  accurately  and  adequately  budget  for  the 
program,  make  the  EVM  systems  useful  for  program  oversight,  and  arrive 
at  reliable  cost  estimates. 

Finally,  from  a  management  perspective,  CMA  could  improve  its  use  of 
award  fees  to  incentivize  the  systems  contractors  to  try  to  meet  the  CWC 
treaty  deadline  in  a  safe  and  environmentally  compliant  manner, 
particularly  at  the  incineration  sites  where  award  fees  are  predominant. 

We  identified  opportunities  for  CMA  to  continue  and  further  advance  its 
recent  efforts  to  emphasize  schedule  and  cost  and  to  incorporate  more 
objective  measures  in  its  award  fee  evaluation  criteria  as  called  for  in  DOD 
guidance  on  award  fees.  Given  the  maturity  of  the  operations  phase,  the 
experience  gained  by  the  systems  contractors,  and  the  composition  of  the 
remaining  stockpile,  CMA  may  be  able  to  adjust  the  award  fee  plan  to 
provide  better  interim  incentives  to  accelerate  the  systems  contractors’ 
performance,  and  do  so  in  a  safe  and  environmentally  sound  manner. 
Doing  so  would  also  have  the  benefit  of  reducing  the  security 
vulnerabilities  and  other  risks  of  storing  lethal  chemical  agents  any  longer 
than  necessary,  which  creates  additional  safety  and  environmental 
hazards.  We  recognize  that  to  the  extent  that  objective  criteria  are  not 
feasible,  using  subjective  criteria  to  determine  award  fees  is  reasonable 
and  expected,  and  DOD  guidance  directs  that  award  fee  criteria  be  tied  to 
performance  measures,  such  as  identifiable  interim  outcomes,  discrete 
events,  or  milestones,  as  much  as  possible.  We  identified  opportunities  for 
CMA  to  improve  the  effectiveness  of  its  monetary  incentives  by  better 
aligning  its  award  fee  policy  and  evaluation  criteria  to  DOD’s  March  2006 
guidance. 
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Recommendations  for 
Executive  Action 


We  are  making  13  recommendations  for  executive  action.  They  are  as 
follows. 

To  strengthen  program  management  and  increase  the  likelihood  that  more 
progress  can  be  made  in  destroying  chemical  agents,  we  recommend  that 
the  Secretary  of  Defense  direct  the  Secretary  of  the  Army  to  take  the 
following  three  actions: 

incorporate  baseline  and  trend  data  for  past  performance  and  multiyear 
performance  goals  for  the  future  in  its  annual  performance  plan; 

develop  interim  destruction  goals,  approaches,  and  milestones  that  are 
directly  linked  to  overall  program  goals  for  meeting  CWC  deadlines;  and 

establish  a  time  frame  for  completing  and  implementing  its  risk 
management  approach,  including  integration  across  sites  and  with  DOD. 

To  provide  more  realistic  cost  estimates  for  chemical  demilitarization  sites 
in  light  of  recent  experience  and  information  gained,  we  recommend  that 
the  Secretary  of  Defense  direct  the  Secretary  of  the  Army  to  update  the 
2005  program  schedule  milestones  for  each  of  DOD’s  chemical 
demilitarization  sites  to  reflect  recent  site  processing  rates  and  then  adjust 
the  cost  estimate  accordingly. 

To  help  ensure  that  the  program  baseline  schedule  is  current  and 
achievable  and  to  improve  the  accuracy  and  reliability  of  program 
manager  estimates  for  measuring  the  Chemical  Demilitarization  Program’s 
progress,  we  recommend  that  the  Secretary  of  Defense  direct  the  CAIG  to 
establish  a  periodic  schedule  to  review  the  program’s  processing  history  to 
determine  whether  the  schedule  milestones  are  still  reasonable. 

To  increase  the  ability  of  decision  makers  to  measure  the  Army’s  and 
CMA’s  performance  against  schedule  growth,  we  recommend  that 
Secretary  of  Defense  direct  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology,  and  Logistics  to  include  in  the  Chemical 
Demilitarization  Program’s  APB  and  SAR  the  updated  and  reviewed 
schedule  milestones  for  completing  destruction  operations  and  facility 
closure  activities  for  each  of  the  chemical  agent  destruction  sites. 

To  provide  more  realistic  cost  estimates  for  closure  of  chemical 
demilitarization  sites  and  to  establish,  stabilize,  and  maintain  an  accurate, 
valid,  and  current  performance  management  baseline,  we  recommend  that 
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the  Secretary  of  Defense  direct  the  Secretary  of  the  Army  to  take  the 
following  three  actions: 

•  coordinate  with  the  systems  contractors  in  defining  closure  requirements 
and  developing  accurate  and  realistic  closure  cost  estimates  for  each  of 
DOD’s  chemical  demilitarization  sites’  contract  performance  measurement 
baseline; 

•  define  all  contract  requirements,  including  authorized  unpriced  work 
associated  with  facility  closure  activities,  and  develop  comprehensive  cost 
estimates  through  closure  at  each  chemical  destruction  site;  and 

•  establish  a  time  frame  for  completing  the  negotiation  for  placing  all 
authorized  work  that  has  not  been  priced  on  contract  and  ensuring  that 
the  negotiations  take  place. 

After  the  cost  estimates  are  complete  and  schedule  milestones  have  been 
updated  and  reviewed,  we  recommend  that  the  Secretary  of  Defense  direct 
the  CAIG  to  conduct  an  independent  cost  review  of  the  cost  estimates. 

To  continue  to  further  the  Army’s  efforts  to  improve  the  potential  to  meet 
the  CWC  deadline  for  destroying  chemical  weapons  safely  and  within 
environmental  compliance  and  reduce  the  risk  of  storing  lethal  chemical 
agents  any  longer  than  necessary,  we  recommend  that  the  Secretary  of  the 
Defense  direct  the  Secretary  of  the  Army  to  determine  for  each  of  the  four 
incineration  site  contracts  whether  greater  weight  can  be  applied  to 
performance  measures,  such  as  schedule  and  cost,  in  award  fee  plans  to 
further  incentivize  the  systems  contractors  to  increase  the  destruction  rate 
of  chemical  agents  in  a  safe  and  environmentally  sound  manner. 

To  further  link  monetary  incentives  and  key  acquisition  outcomes,  and  to 
provide  a  means  for  objectively  measuring  the  systems  contractors’ 
performance  during  the  assessment  period  as  much  as  possible,  we 
recommend  that  the  Secretary  of  Defense  direct  the  Secretary  of  the  Army 
to  take  the  following  two  actions: 

•  better  align  its  award  fee  policy  with  DOD’s  March  2006  guidance  on 
award  fees  and 

•  link  more  of  its  award  fee  criteria  to  performance  measures  that  focus  on 
identifiable  interim  outcomes,  discrete  events,  or  milestones. 
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Agency  Comments 
and  Our  Evaluation 


In  written  comments  on  a  draft  of  this  report,  DOD  concurred  or  partially 
concurred  with  12  of  our  13  recommendations.  DOD  did  not  concur  with 
our  recommendation  regarding  defining  closure  requirements  and 
developing  accurate  and  realistic  closure  cost  estimates  for  each  of  DOD’s 
chemical  demilitarization  sites.  DOD  recommended  that  we  delete  it,  since 
DOD  officials  believe  that  a  subsequent  recommendation  defining  all 
contract  requirements,  including  those  associated  with  facility  closure 
activities,  was  more  encompassing.  Based  on  DOD’s  comments,  rather 
than  deleting  the  recommendation,  we  clarified  it  and  the  related 
recommendation  to  more  clearly  distinguish  the  difference  between  the 
two  recommendations,  and  consolidated  it  with  our  two  other 
recommendations  related  to  defining  contract  requirements.  DOD  also 
stated  that  the  department  will  continue  to  use  acquisition  management 
tools  and  discipline  to  ensure  the  destruction  of  the  United  States’ 
chemical  weapons  stockpile  in  a  safe  and  secure  manner,  while  being 
economical  and  meeting  the  U.S.  commitments  under  the  Chemical 
Weapons  Convention.  DOD  identified  a  number  of  actions  that  it  already 
has  initiated  in  response  to  our  recommendations.  DOD  also  provided  us 
with  technical  comments,  which  we  incorporated  in  the  report,  as 
appropriate.  DOD’s  comments  are  reprinted  in  appendix  III. 

Regarding  our  four  recommendations  related  to  strategic  planning  tools, 
DOD  concurred  with  our  recommendation  that  it  incorporate  baseline  and 
trend  data  for  past  performance  and  multiyear  performance  goals  for  the 
future  in  its  annual  performance  plan.  DOD  stated  that  CMA  currently  is 
incorporating  these  items  into  its  fiscal  year  2008  Annual  Performance 
Plan  and  expects  this  effort  to  be  completed  by  December  31,  2007.  DOD 
partially  concurred  with  our  recommendation  that  it  provide  direction  to 
develop  interim  destruction  goals,  approaches,  and  milestones  that  are 
directly  linked  to  overall  program  goals  for  meeting  CWC  deadlines.  In  the 
spirit  of  our  recommendation,  DOD  stated  that  it  would  ensure  that  all 
future  CMA  performance  plans  incorporate  annual  goals  and  approaches 
leading  toward  the  April  29,  2012,  destruction  deadline  through  direction 
issued  annually  in  support  of  its  existing  strategic  governance  process  and 
that  additional  direction  would  not  be  needed.  We  support  this  planned 
action,  which,  if  implemented  properly,  would  accomplish  the  same  effect 
as  our  recommendation.  DOD  concurred  with  our  recommendation  that 
the  Army  establish  a  time  frame  for  completing  and  implementing  its  risk 
management  approach,  including  integration  across  sites  and  with  DOD. 
DOD  stated  that  it  expects  an  approved  risk  management  approach  to  be 
completed  by  June  30,  2008.  DOD  concurred  with  our  recommendation  to 
include  in  the  Chemical  Demilitarization  Program’s  APB  and  SAR  the 
updated  and  reviewed  schedule  milestones  for  completing  destruction 
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operations  and  facility  closure  activities  for  each  of  the  chemical  agent 
destruction  sites.  DOD  stated  that  it  would  request  administrative  changes 
to  remove  placeholder  terminology  from  these  documents  and  replace 
them  with  the  CAIG’s  schedule  estimates. 

Regarding  our  two  recommendations  related  to  schedule  and  cost 
estimates,  DOD  partially  concurred  with  our  recommendation  that  it 
update  the  2005  program  schedule  milestones  for  each  of  DOD’s  chemical 
demilitarization  sites  to  reflect  recent  site  processing  rates  and  then  adjust 
the  cost  estimate  accordingly.  DOD  stated  that  it  has  gained  only  limited 
information  during  the  past  year  regarding  processing  rates,  unknown 
risks,  and  the  potential  for  a  chemical  event,  and  thus  does  not  believe  it 
would  be  prudent  to  adjust  cost  and  schedule  estimates  or  rebaseline  at 
this  time.  However,  DOD  will  evaluate  the  cost  and  schedule  estimates  for 
the  entire  program  during  the  upcoming  Fiscal  Year  2010-2015  Program 
Objectives  Memorandum  build  process  and  program  review,  and  will 
determine  whether  an  update  is  warranted.  DOD  concurred  with  our 
recommendation  to  establish  a  periodic  schedule  for  the  CAIG  to  review 
the  program’s  processing  history  to  determine  whether  the  schedule 
milestones  are  still  reasonable.  DOD  stated  that  it  conducts  a  biannual 
program  review  of  the  Chemical  Demilitarization  Program  and  that  during 
the  next  review  in  the  spring  of  2008,  it  would  evaluate  the  program’s  cost 
and  schedule  estimates.  If  significant  differences  exist  between  the 
program’s  baseline  and  the  revised  estimates,  then  DOD  would  request  a 
CAIG  review  that  could  result  in  new  cost  and  schedule  estimates. 
However,  we  believe  that  our  report  provides  clear  and  sufficient  evidence 
demonstrating  that  there  are  differences  between  the  program’s  current 
schedule  and  cost  estimates  and  DOD’s  baseline  to  warrant  a  CAIG  review 
and  update  as  we  recommended. 

Regarding  our  four  recommendations  related  to  contract  requirements, 
based  on  DOD’s  comments,  we  consolidated  and  clarified  two  of  our 
recommendations.  DOD  suggested  that  we  delete  the  recommendation 
that  CMA  coordinate  with  the  systems  contractors  in  defining  closure 
requirements  and  in  developing  accurate  and  realistic  closure  cost 
estimates  for  each  of  DOD’s  chemical  demilitarization  sites,  since  the 
subsequent  recommendation  is  more  encompassing.  Instead,  we  clarified 
this  recommendation  by  adding  “for  each  of  DOD’s  chemical 
demilitarization  sites’  contract  performance  measurement  baseline" 
[emphasis  added],  and  consolidated  it  with  our  two  other 
recommendations  related  to  defining  contract  requirements  discussed 
below.  As  discussed  in  our  report,  we  continue  to  believe  that  CMA  and 
the  systems  contractors  must  coordinate  their  efforts  to  develop  accurate 
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and  realistic  closure  cost  estimates.  In  addition,  we  consolidated  our 
recommendation  that  after  the  cost  estimates  are  complete  and  schedule 
milestones  have  been  updated  and  reviewed,  the  CAIG  conduct  an 
independent  cost  review  of  the  cost  estimates  with  our  two 
recommendations  that  are  discussed  below.  DOD  partially  concurred  with 
our  recommendation  to  define  all  contract  requirements,  including  those 
associated  with  facility  closure  activities,  and  develop  comprehensive  cost 
estimates  through  closure  at  each  chemical  destruction  site.  Based  on 
DOD’s  comments,  we  clarified  this  recommendation  by  adding  “define  all 
contract  requirements,  including  authorized  unpriced  work  associated 
with  facility  closure  activities”  [emphasis  added],  DOD  stated  that  since 
June  2007,  three  of  the  five  CMA  destruction  sites  have  completed  contract 
performance  baselines  that  include  contract  requirements  for  operations, 
secondary  waste  processing,  closure,  and  known  risk.  Negotiations  are 
under  way  to  complete  the  contract  performance  baseline  for  the  Tooele, 
Utah,  site  and  the  contract  requirements  and  cost  estimates  for  the 
Newport,  Indiana,  site  by  June  30,  2008.  DOD  concurred  with  our 
recommendation  to  establish  a  time  frame  for  completing  the  negotiation 
for  placing  all  authorized  work  that  has  not  been  priced  on  contract  and 
ensuring  that  the  negotiations  take  place.  DOD  stated  that  it  expects 
negotiations  to  be  completed  by  June  30,  2008.  Finally,  DOD  partially 
concurred  with  our  recommendation  that  the  CAIG  conduct  an 
independent  review  of  the  program’s  cost  estimates  stating  that  the 
program’s  biannual  review  in  support  of  the  budget  submission  will 
determine  if  a  CAIG  independent  assessment  is  warranted.  We  continue  to 
believe  that  providing  an  independent  review  of  the  program’s  cost 
estimates  could  provide  greater  assurance  that  cost  estimates  are 
reasonable. 

With  regard  to  our  three  recommendations  related  to  incentive  fees,  DOD 
concurred  with  our  recommendation  to  determine  for  each  of  the  four 
incineration  site  contracts  whether  greater  weight  can  be  applied  to 
performance  measures,  such  as  schedule  and  cost,  in  award  fee  plans  to 
further  incentivize  the  systems  contractors  to  increase  the  destruction  rate 
of  chemical  agents  in  a  safe  and  environmentally  sound  manner.  DOD  also 
concurred  with  our  recommendation  to  better  align  CMA’s  award  fee 
policy  with  DOD’s  March  2006  guidance  on  award  fees,  stating  that  this 
action  had  been  completed  on  September  7,  2007.  However,  when  a  copy 
of  the  revised  policy  was  requested,  DOD  stated  that  its  response 
incorrectly  referenced  that  this  action  has  been  completed.  As  such,  DOD 
will  direct  a  revision  of  the  CMA  policy  that  aligns  it  with  the  DOD 
guidance  on  award  fees,  and  the  action  is  to  be  completed  by  March  31, 
2008.  DOD  partially  concurred  with  our  recommendation  to  link  more  of 
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its  award  fee  criteria  to  performance  measures  that  focus  on  identifiable 
interim  outcomes,  discrete  events,  or  milestones.  DOD  stated  that  it  would 
conduct  a  review  of  CMA  award  fee  plans,  to  be  completed  by  March  31, 
2008,  and  would  then  determine  whether  further  direction  and  action  is 
required.  While  this  review  represents  a  positive  step,  our  report  shows 
that  there  are  several  areas  in  which  DOD  could  better  link  more  of  its 
award  fee  criteria  with  performance  measures  that  focus  on  identifiable 
outcomes  and  we  believe  that  DOD  should  fully  implement  this 
recommendation. 


As  agreed  with  your  offices,  unless  you  publicly  announce  the  contents  of 
this  report  earlier,  we  plan  no  further  distribution  of  it  until  30  days  from 
the  date  of  this  letter.  We  will  then  send  copies  of  this  report  to  the 
Chairmen  and  Ranking  Members  of  the  Senate  Committee  on  Armed 
Services;  Senate  Committee  on  Homeland  Defense  and  Governmental 
Affairs;  Subcommittee  on  Defense,  Senate  Committee  on  Appropriations; 
House  Committee  on  Oversight  and  Government  Reform;  and 
Subcommittee  on  Defense,  House  Committee  on  Appropriations,  and 
other  interested  congressional  parties.  We  are  also  sending  copies  of  this 
report  to  the  Secretary  of  Defense;  the  Secretary  of  the  Army;  the  Director, 
Office  of  Management  and  Budget;  and  the  Director,  U.S.  Army  Chemical 
Materials  Agency.  We  also  will  make  copies  available  to  others  on  request. 
In  addition,  this  report  will  be  available  at  no  charge  on  the  GAO  Web  site 
at  http://www.gao.gov. 

If  you  or  your  staff  have  any  questions  concerning  this  report,  please 
contact  me  at  (202)  512-5431  or  dagostinod@gao.gov.  Contact  points  for 
our  Offices  of  Congressional  Relations  and  Public  Affairs  may  be  found  on 
the  last  page  of  this  report.  GAO  staff  who  made  key  contributions  to  this 
report  are  listed  in  appendix  IV. 


Davi  M.  D’Agostino 
Director,  Defense  Capabilities 
and  Management 
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Appendix  I:  Scope  and  Methodology 


To  assess  the  progress  the  Department  of  Defense  (DOD)  and  the  Army 
have  made  in  implementing  our  prior  recommendations  to  strengthen 
program  management,  whether  it  has  been  sufficient,  and  whether  actions 
taken  to  identify  and  mitigate  the  risk  of  future  program  schedule 
extensions  and  cost  growth  have  been  effective,  we  obtained  and  reviewed 
strategic  and  implementation  plans  and  interviewed  cognizant  officials  to 
identify  whether  key  elements  were  present,  such  as  a  program  mission 
statement,  long-term  goals  and  objectives,  delineation  of  roles  and 
responsibilities  of  DOD  and  Army  offices,  and  near-term  performance 
measures. 

To  assess  the  reasonableness  of  the  methodology  used  to  determine 
schedule  milestones  as  well  as  the  reliability  of  cost  estimates,  we 
reviewed  current  program  estimates,  destruction  schedules,  earned  value 
management  (EVM)  data,  and  other  documents.  We  also  obtained  and 
reviewed  the  program’s  risk  management  plans  and  related  documents.  In 
addition,  we  also  identified  the  issues  that  had  caused  delays  and 
ascertained  approaches  being  used  to  reduce  the  potential  for  delays  in 
the  future. 

The  Army’s  Chemical  Materials  Agency  (CMA)  provided  us  with  EVM 
reports  submitted  to  the  Army  by  the  systems  contractors  for  each  of  the 
chemical  demilitarization  facilities  spanning  the  period  from  December 
2000  through  February  2007.  EVM  reports  were  provided  electronically  in 
the  form  of  Deltek  wlnsight™  transfer  files  for  Cost  Performance  Report 
Formats  1  through  4  and  separate  Adobe  ’,  MS  Word,  or  MS  Excel  files  for 
Format  5. 

Our  approach  in  assessing  the  adequacy  of  the  systems  contractors’  EVM 
reporting  was  to  analyze  the  data  provided  according  to  EVM  analysis 
tasks  and  DOD  earned  value  implementation  guidance.  The  GAO  EVM 
analysis  tasks  are  a  set  of  10  analysis  tasks  developed  by  GAO  that  are 
used  to  assess  the  adequacy  of  EVM.  These  10  analysis  tasks  were 
performed  on  each  of  the  EVM  reports  provided  to  us.  Guidance  and 
procedures  outlined  within  the  DOD  Earned  Value  Management 
Implementation  Guide  were  also  referenced  to  assess  EVM  system  and 
reporting  adequacy. 

Our  approach  in  assessing  the  Army’s  ability  to  assess,  synthesize,  and 
incorporate  systems  contractor  historical  and  estimated  cost,  technical, 
and  schedule  performance  into  the  current  program  estimate  was  to  align 
historical  contract  performance  data  as  documented  within  the  EVM 
reports  with  program  estimate  and  baseline  data  documented  within 
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Selected  Acquisition  Reports,  the  President’s  budget,  and  the  Fiscal  Year 
2005  Army  Cost  Position. 

In  performing  this  assessment,  we  also  conducted  interviews  with 
government  and  systems  contractor  officials  at  CMA  headquarters  in 
Aberdeen,  Maryland;  Tooele  Chemical  Agent  Disposal  Facility,  Deseret 
Chemical  Depot  (Tooele),  Utah;  and  Umatilla  Chemical  Agent  Disposal 
Facility,  Umatilla  Chemical  Depot  (Umatilla),  Oregon. 

To  analyze  the  effectiveness  of  DOD’s  and  the  Army’s  use  of  monetary 
incentives  to  improve  systems  contractors’  performance,  we  obtained  and 
reviewed  contract  documentation  related  to  the  implementation  of  the 
contracts’  incentives,  including  the  basic  contract  and  modifications, 
award  fee  plans,  award  fee  determining  official  decisions,  award  fee 
evaluation  board  reports,  and  pre-  and  postnegotiation  memorandums, 
and  the  performance-based  incentive  fees,  including  the  Director’s 
Programmatic  Performance  Based  Incentive  (DPPBI)  fee.  We  also 
interviewed  DOD  and  Army  officials  to  determine  the  types  of  award  and 
incentive  fees  offered  at  the  incineration  sites  and  at  Newport  Chemical 
Agent  Disposal  Facility,  Newport  Chemical  Depot  (Newport),  Indiana. 

To  evaluate  the  effectiveness  of  the  award  fees  in  controlling  costs  and 
schedule,  we  reviewed  the  award  fee  plans  from  fiscal  year  2003  to  fiscal 
year  2007  to  identify  the  types  of  performance  measures  and  weight 
factors  used  to  incentivize  the  systems  contractors  to  safely  destroy  the 
chemical  weapons  stockpile.  We  also  looked  at  the  extent  the  criteria  used 
for  the  performance  measures  were  linked  to  measurable  outcomes.  To 
identify  the  percentage  of  award  fees  paid  to  the  systems  contractors  at 
the  incineration  sites  and  Newport,  we  collected  and  reviewed  data  on  the 
award  fee  payments,  including  the  amount  available  for  the  contactors  to 
earn,  the  amounts  earned  from  fiscal  year  2003  through  fiscal  year  2006, 
and  the  guidance  for  improving  the  use  of  these  fees.  We  used  data  on  the 
award  fee  plans  from  fiscal  year  2003  through  fiscal  year  2007  because 
they  represented  the  period  of  time  the  sites  were  in  the  operations  phase. 

To  evaluate  the  extent  documentation  was  prepared  for  adjustments  to  the 
award  fee  plans,  we  reviewed  CMA’s  Integrated  Process  Team  documents 
from  fiscal  year  2003  to  fiscal  year  2007  to  identify  adjustments  made  by 
the  various  stakeholders,  such  as  the  CMA  Director,  U.S.  Army 
Sustainment  Command,  CMA  Program  and  Policy  Office,  incinerator  site 
project  managers,  and  systems  contractor  site  general  managers.  We 
evaluated  whether  Army  officials  provided  complete  documentation 
supporting  their  rationale  for  adjusting  performance  measures  and  related 
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weights  throughout  the  process.  We  also  evaluated  whether  CMA  prepared 
adequate  supporting  documentation  to  enable  reviewers  to  understand 
and  evaluate  the  process  that  was  used,  and  whether  it  contained,  among 
other  things,  an  adequate  discussion  of  alternative  performance  measures 
or  the  combination  of  award  and  incentive  fees  that  it  considered,  tested, 
and  rejected  and  the  reasons  for  rejecting  them. 

To  evaluate  the  effectiveness  of  the  authority  contained  in  section  923  of 
the  National  Defense  Authorization  Act  for  Fiscal  Year  2007,  we  obtained 
and  reviewed  the  incentive  agreement  plan  and  incentive  fees  for  the 
Aberdeen  Chemical  Agent  Disposal  Facility  (Aberdeen),  Maryland,  and 
Newport  to  determine  the  challenges  those  sites  faced  in  negotiating  the 
incentive  fee  agreements.  Also,  we  interviewed  officials  with  the  U.S. 

Army  Sustainment  Command,  CMA,  and  CMA  and  systems  contractors  at 
Tooele  and  Umatilla. 

We  also  met  with  DOD  and  Army  program  officials  and  interviewed 
officials  at  two  of  the  chemical  agent  destruction  sites.  Finally,  this  report 
focuses  primarily  on  the  seven  CMA-managed  chemical  destruction  sites — 
Aberdeen;  Anniston  Chemical  Agent  Disposal  Facility,  Anniston  Chemical 
Activity,  Alabama;  Johnston  Atoll  in  the  Pacific  Ocean;  Newport;  Pine 
Bluff  Chemical  Agent  Disposal  Facility,  Pine  Bluff  Chemical  Activity, 
Arkansas;  Tooele;  and  Umatilla — because  most  are  in  the  operational 
phase  and  represent  about  82  percent  of  the  remaining  stockpile  of 
chemical  agents  to  be  destroyed,  and  have  reached  a  level  of  maturity  in 
processing  a  variety  of  types  of  munitions  and  agents  that  will  help  meet 
Chemical  Weapons  Convention  (CWC)  goals.  This  report  does  not  focus 
on  two  sites — Aberdeen  and  Johnston  Atoll — that  have  already  completed 
operations  and  have  either  been  closed  or  are  being  closed.  It  also  does 
not  focus  on  the  two  chemical  agent  destruction  sites  that  are  part  of  the 
Assembled  Chemical  Weapons  Alternatives  program — Blue  Grass 
Chemical  Agent-Destruction  Pilot  Plant,  Blue  Grass  Chemical  Activity 
(Blue  Grass),  Kentucky,  and  Pueblo  Chemical  Agent-Destruction  Pilot 
Plant,  Pueblo  Chemical  Depot,  Colorado — that  currently  are  in  the  design 
and  construction  phase  to  destroy  chemical  agents,  report  directly  to  DOD 
rather  than  to  the  Army,  and  are  not  expected  to  help  meet  CWC  goals. 

We  examined  the  reliability  of  the  data  used  in  this  report  by  verifying 
EVM  system  certification,  applying  analytical  tests  to  the  EVM  data,  and 
examining  EVM  system  effectiveness  in  providing  meaningful  performance 
measurement  to  program  management.  We  found  that  EVM  systems  at 
each  site  had  been  certified.  The  results  of  our  analytical  tests  found  the 
data  to  be  reasonably  free  of  data  errors  and  anomalies.  However,  when 
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we  examined  the  effectiveness  of  the  EVM  system  to  provide  meaningful 
performance  measurement,  we  found  insufficient  reliability  to  address  our 
objectives. 

We  conducted  this  performance  audit  from  May  2006  through  July  2007  in 
accordance  with  generally  accepted  government  auditing  standards. 
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Appendix  II:  Two  Recently  Introduced 
Performance-Based  Incentive  Fees 


The  Army’s  performance-based  incentive  fee  to  collaboratively  improve 
schedule  and  cost  performance  was  established  in  2005  because  the  use  of 
award  fees  had  not  been  successful.  Also,  the  Army  plans  to  use  a  recently 
authorized  $165  million  incentive  fee  to  accelerate  destruction  and  closure 
activities  to  dates  linked  with  the  treaty  schedule. 


Army’s  Performance-Based 
Incentive  Fee  Begun  in 
2005  Was  Recently 
Targeted  to  Objective 
Performance 


In  2005,  CMA  established  the  DPPBI  fee  to  improve  schedule  and  cost 
performance  because  CMA’s  use  of  award  fees  had  not  been  successful  in 
controlling  cost  and  schedule  at  the  incineration  sites.  This  incentive  fee  is 

2  percent  of  the  total  12  percent  available  fees,  which  includes  the 

3  percent  base  fee  that  is  not  performance  related.  DPPBI  was  structured 
so  that  the  collective  performance  of  the  systems  contractors  is  evaluated 
and  rewarded  through  improved  collaboration  among  the  systems 
contractors  that  operate  the  four  incineration  sites.  That  is,  while  each 
individual  site  is  evaluated  and  scored  separately,  the  individual  scores  are 
averaged  to  determine  a  composite  score  that  results  in  all  sites  earning 
the  same  percentage  of  the  available  DPPBI  fee  pool. 


The  evaluation  factors  for  assessing  the  systems  contractor  performance 
for  DPPBI  have  changed  over  time  and  are  divided  between  objective 
criteria  and  a  subjective  assessment  of  systems  contractor  performance 
during  the  fiscal  year.  Three  criteria  were  established  for  the  initial  DPPBI 
evaluation  period  of  January  1  through  September  30,  2005:  risk 
mitigation,  production  assessment,  and  lessons  learned.  In  the  initial 
evaluation  period,  the  contractors  collectively  were  deemed  to  have 
earned  51  percent  of  the  2005  DPPBI  pool  or  $3,313,954  out  of  $6,497,949. 
In  fiscal  year  2006,  the  DPPBI  criteria  were  changed  to  put  more  emphasis 
on  performance  outputs  and  less  on  processes  and  procedures.  The  fiscal 
year  2006  DPPBI  criteria  were  established  to  encourage  the  contractors  to 
reduce  project  schedules  while  maintaining  a  high  standard  of  surety, 
safety,  and  environmental  compliance.  The  criteria  also  were  established 
to  increase  CMA’s  confidence  levels  that  the  sites  could  achieve  CWC 
destruction  deadlines,  although  discrete  goals  were  not  specified.  For  the 
fiscal  year  2006  period  of  performance,  the  contractors  earned  about 
93  percent  or  $8,771,200  out  of  the  $9,411,159  in  the  DPPBI  pool  for  that 
period. 

Until  fiscal  year  2007,  the  DPPBI  fee  evaluation  was  not  linked  to  objective 
schedule  performance  criteria  linked  to  the  April  29,  2012,  CWC  extended 
treaty  deadline.  In  fiscal  year  2007,  schedule  performance  attainment  was 
added  to  the  DPPBI  objective  criteria  amounting  to  70  percent  of  the  total 
available  DPPBI  fee  (or  up  to  1.4  percent  for  average  collective 
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performance)  and  amounting  to  30  percent  for  subjective  criteria  related 
to  the  contractors’  performance  in  the  application  of  lessons  learned,  risk 
management,  programmatic  procurement,  and  other  collaborative 
approaches  to  reduce  cost  and  accelerate  destruction  of  the  U.S.  stockpile. 
The  amount  of  the  fee  that  can  be  earned  by  the  systems  contractors 
depends  not  just  on  a  systems  contractor’s  individual  performance,  but 
also  on  the  performance  of  the  other  systems  contractors.  For  example,  if 
the  systems  contractor  at  Anniston  is  performing  well  in  meeting  its 
schedule-related  goals  during  a  specific  period,  but  the  systems  contractor 
at  Tooele  is  not,  then  the  Anniston  systems  contractor  would  be 
encouraged  to  offer  assistance  to  the  Tooele  systems  contractor.  If  the 
Tooele  systems  contractor’s  schedule  performance  could  not  be  improved, 
then  the  Anniston  systems  contractor’s  potential  to  earn  this  fee  would  be 
reduced  as  would  the  potential  of  systems  contractors  at  the  other  sites. 


Recently  Authorized 
Incentives  Are  Targeted  to 
Completing  Destruction 
Operations  around  the 
Treaty  Deadline  and 
Accelerating  Facility 
Closure  Activities 


CMA  plans  to  offer  incentive  fees  under  the  authority  contained  in  section 
923  of  the  National  Defense  Authorization  Act  for  Fiscal  Year  2007, 1 
permitting  systems  contractors  the  opportunity  to  earn  incentive  payments 
for  the  completion  of  destruction  operations  by  or  near  the  CWC  deadline 
and  the  acceleration  of  facility  closure  activities,  but  the  effectiveness  of 
this  approach  is  uncertain.  The  purpose  of  section  923  is  to  provide  the 
systems  contractor  for  a  chemical  demilitarization  facility  an  incentive  to 
accelerate  the  safe  elimination  of  the  U.S.  chemical  weapons  stockpile  and 
to  reduce  the  total  cost  of  the  Chemical  Demilitarization  Program  by 
providing  incentive  payments  for  the  early  completion  of  destruction 
operations  and  the  closure  of  such  a  facility.2  Section  923  authorizes  the 
inclusion  of  incentives  clauses  in  chemical  demilitarization  contracts  for 
the  completion  of  both  destruction  operations  and  facility  closure 
activities  within  target  incentive  ranges.  Target  incentive  ranges  are  to  be 
jointly  agreed  upon  by  the  contracting  officer  and  the  systems  contractor 
concerned  and  are  to  be  specified  in  the  contract’s  incentives  clause. 

Under  the  statute,  the  amount  of  incentive  payment  a  systems  contractor 
may  earn  through  such  incentive  clauses  will  be  based  on  how  early  within 
the  specified  target  incentive  range  the  systems  contractor  completes 
destruction  operations  and  facility  closure  activities.  Section  923  sets 
limitations  on  the  amount  of  incentive  payments  a  systems  contractor  may 
receive  through  the  incentives  clauses.  It  establishes  $110  million  as  the 


'Pub.  L.  No.  109-364  (2006). 

2Pub.  L.  No.  109-364,  §  923  (a)(2)  (2006). 
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maximum  incentive  payment  for  the  completion  of  destruction  operations 
within  the  target  incentive  range  specified  under  the  contract’s  incentives 
clause  and  $55  million  for  the  completion  of  facility  closure  activities 
within  the  target  incentive  range  specified  under  the  contract’s  incentives 
clause.3 

Although  it  had  not  been  implemented  at  the  time  of  our  review,  CMA  had 
drafted  a  plan  for  implementation  of  the  incentives  authorized  under 
section  923.4  The  plan  articulates  that  the  systems  contractors  are  to 
assume  responsibility  for  identifying  and  mitigating  all  site-specific  and 
programmatic  risks  in  order  to  successfully  complete  program  objectives 
within  budgeted  cost  and  schedule  targets.  The  plan  further  relates  that  in 
return,  the  systems  contractors  will  receive  incentives  measured  against 
established  schedule  targets  for  operations  completion  and  facility 
closure. 

Under  CMA’s  plan,  it  would  offer  the  additional  incentive  fee  of  up  to 
$150  million  to  each  site — $100  million  for  destruction  operations 
completed  within  target  ranges  by  or  near  the  April  29,  2012,  treaty 
deadline,  and  $50  million  for  facility  closure  activities  brought  successfully 
to  completion  within  target  ranges.  CMA’s  plan  establishes  the  period  from 
November  1,  2011,  through  October  31,  2013,  as  the  target  incentive  range 
for  the  completion  of  operations  at  all  sites,  with  the  maximum  amount 
available  to  be  earned,  exclusive  of  the  bonus,  at  $100  million.  Ninety-five 
percent  of  the  incentive  can  be  earned  for  completing  operations  by  the 
CWC  extended  deadline  of  April  29,  2012.  After  the  CWC  deadline,  a 
percentage  of  the  incentives  can  be  earned  through  October  31,  2013,  on  a 
reduced  sliding  scale.  The  CMA  plan  also  offers  up  to  $50  million  per  site 
for  the  completion  of  facility  closure  activities  within  a  specified  range  of 
dates  unique  to  each  site.  One  hundred  percent  of  the  $50  million  would  be 
earned  by  a  systems  contractor  for  completion  of  facility  closure  activities 
on  the  first  date  of  the  specified  range.  The  systems  contractor  would  earn 
some  lesser  percentage  of  the  $50  million  for  completion  on  any  date 
between  the  first  and  last  dates  of  the  specified  range. 


3Pub.  L.  No.  109-364,  §  923  (b)(2)  (2006). 

4Chemical  Materials  Agency,  Incentive  Agreement  for  Completion  of  Destruction 
Operations  and  Facility  Closure  Activities  at  Chemical  Stockpile  Disposal  Facilities 
(Washington,  D.C.:  May  19,  2007). 
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Under  CMA’s  plan,  systems  contractors  can  earn  up  to  $165  million  at  their 
respective  sites  through  bonus  potential.  In  addition  to  the  potential 
$100  million  fee  for  completing  operations  ahead  of  the  extended  CWC 
deadline  and  $50  million  for  meeting  specified  closure  targets,  bonuses 
can  also  be  earned.  A  bonus  of  $10  million  is  offered  to  each  site  if  all  the 
sites  complete  destruction  operations  prior  to  the  CWC  deadline  of  April 
29,  2012.  A  bonus  of  $5  million  is  also  offered  to  each  site  for  accelerated 
completion  of  facility  closure  activities. 

CMA’s  plan,  which  would  have  to  be  implemented  under  each  of  the 
contracts,  includes  language  that  insulates  the  government  from  having  to 
make  incentive  payments  to  the  systems  contractors  if  they  did  not  strictly 
perform  under  the  terms  CMA  has  set  under  the  plan.  In  firmly 
establishing  the  target  incentive  ranges,  the  plan  states  that  the  target 
dates  are  not  subject  to  extension  except  for  catastrophic  causes  beyond 
the  control  of  and  without  the  fault  or  negligence  of  the  systems 
contractor,  its  subcontractors,  and  suppliers.  The  plan  establishes  that  the 
systems  contractors  are  not  entitled  to  submit  claims  for  loss  of 
opportunity  to  earn  the  incentives  or  to  increase  incentive  pools  due  to 
failure  of  the  government  to  provide  resources  over  and  above  those 
specifically  identified  in  the  contract.  It  also  stipulates  that  payments  of 
incentives  will  be  subject  to  adjustment  based  upon  final  audits  of 
operation  costs  and  closure  costs.  At  the  same  time,  the  plan  recognizes, 
as  does  the  statute,  that  the  enterprise  of  striving  to  capture  these 
incentives  is  to  be  conducted  consistent  with  the  existing  obligation  to 
provide  for  maximum  protection  for  the  environment,  the  general  public, 
and  the  personnel  who  are  involved  in  the  destruction  of  the  chemical 
agents  and  munitions.  The  effectiveness  of  this  planned  incentive  is 
uncertain  because  it  had  not  yet  been  negotiated  or  placed  on  contract  at 
the  time  of  our  review. 
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ASSISTANT  TO  THE  SECRETARY  OF  DEFENSE 

3050  DEFENSE  PENTAGON 
WASHINGTON,  DC  20301-3050 


W  1  8  2007 

Ms.  Davi  M.  D’Agostino 
Director,  Defense  Capabilities  and  Management 
United  States  Government  Accountability  Office 
441  G  Street,  N.W. 

Washington,  DC  20548 

Dear  Ms.  D’Agostino: 

This  is  the  Department  of  Defense  (DoD)  response  to  the  GAO  draft  report, 
GAO-08-134,  ‘CHEMICAL  DEMILITARIZATION:  Additional  Management  Actions 
Needed  to  Meet  Key  Performance  Goals  of  DoD’s  Chemical  Demilitarization  Program,’ 
dated  October  3,  2007  (GAO  Code  350799). 

The  DoD  concurs  with  most  of  the  draft  report’s  recommendations,  except  those 
pertaining  to  when  cost  and  schedule  estimates  should  be  revised.  The  Department  will 
continue  to  use  acquisition  management  tools  and  discipline  to  ensure  the  destruction  of 
the  United  States’  chemical  weapons  stockpile  in  a  safe  and  secure  manner,  while  being 
economical  and  meeting  our  commitments  under  the  Chemical  Weapons  Convention. 

The  Department  appreciates  the  opportunity  to  provide  comments  on  the  draft 
report.  These  comments,  including  technical  comments,  are  enclosed.  For  further 
questions  concerning  this  report,  please  contact  Barbara  Burgess,  Senior  Program  Analyst 
for  the  Chemical  Demilitarization  Program,  (703)  588-1983,  extension  1 13. 

Sincerely, 

Jean  D.  Reed 
Special  Assistant 

Chemical  and  Biological  Defense  and 
Chemical  Demilitarization  Programs 

Enclosures: 

As  stated 

cc: 

Deputy  Assistant  Secretary  of  the  Army  (Elimination  of  Chemical  Weapons) 
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GAO  DRAFT  REPORT  -  DATED  OCTOBER  3, 2007 
GAO  CODE  350799/GAO-08-134 

“CHEMICAL  DEMILTARIZATION:  Additional  Management  Actions  Needed  to  Meet 
Key  Performance  Goals  of  DoD’s  Chemical  Demilitarization  Program” 


DEPARTMENT  OF  DEFENSE  COMMENTS 
TO  THE  RECOMMENDATIONS 


RECOMMENDATION  1:  The  GAO  recommends  the  Secretary  of  Defense  direct  the  Director 
of  U.S.  Army  Chemical  Materials  Agency  to  incorporate  baseline  and  trend  data  for  past 
performance  and  multiyear  performance  goals  for  the  future  in  its  annual  performance  plan. 

DoD  RESPONSE:  Concur.  The  Department  recognizes  that  U.S.  Army  Chemical  Materials 
Agency  (CMA)  is  currently  incorporating  baseline  and  trend  data  for  past  performance  and 
multiyear  performance  measures  in  the  CMA  Annual  Performance  Plan  for  2008.  This  effort  is 
expected  to  be  completed  and  staffed  by  end  of  the  first  quarter  of  Fiscal  Year  2008. 

RECOMMENDATION  2:  The  GAO  recommends  the  Secretary  of  Defense  direct  the  Director 
of  U.S.  Army  Chemical  Materials  Agency  to  develop  interim  destruction  goals,  approaches,  and 
milestones  that  are  directly  linked  to  overall  program  goals  for  meeting  Chemical  Weapons 
Convention  deadlines. 

DoD  RESPONSE:  Partially  Concur.  The  Department  recognizes  that  the  U.S.  Army  Chemical 
Materials  Agency  (CMA)  annual  destruction  goals  are  interim  destruction  goals  for  meeting  the 
Chemical  Weapons  Convention  (CWC)  destruction  milestones.  The  Chemical  Demilitarization 
Program  Fiscal  Year  2007  Annual  Performance  Plan  provided  the  annual  destruction  goals  for 
meeting  the  CWC  45  percent  destruction  milestone  of  December  31,  2007.  The  Department  will 
continue  to  ensure  all  future  CMA  performance  plans  incorporate  annual  goals  and  approaches 
leading  toward  the  April  2012  destruction  deadline  through  direction  issued  annually  in  support 
of  strategic  governance.  Therefore,  no  further  direction  is  required  from  the  Secretary  of 
Defense  to  develop  interim  goals,  approaches,  and  milestones. 

RECOMMENDATION  3:  The  GAO  recommends  the  Secretary  of  Defense  direct  the  Director 
of  U.S.  Army  Chemical  Materials  Agency  to  establish  a  time  frame  for  completing  its  risk 
management  approach,  including  integration  across  sites  and  with  DoD. 

DoD  RESPONSE:  Concur.  The  U.S.  Army  Chemical  Materials  Agency  (CMA)  is  currently 
evaluating  and  updating  the  CMA  risk  management  approach  including  integration  of  risk 
management  across  sites  and  within  CMA  Headquarters.  The  Department  expects  an  approved 
risk  management  approach  to  be  completed  by  the  third  quarter  of  Fiscal  Year  2008. 
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RECOMMENDATION  4:  The  GAO  recommends  the  Secretary  of  Defense  direct  the  Director 
of  U.S.  Army  Chemical  Materials  Agency  to  update  the  2005  program  schedule  milestones  for 
each  of  DoD’s  chemical  demilitarization  sites  to  reflect  recent  site  processing  rates  and  then 
adjust  the  cost  estimate  accordingly. 

DoD  RESPONSE:  Partially  Concur.  The  Department  recognizes  successes  within  the  Chemical 
Demilitarization  Program  (CDP)  over  the  past  two  years.  However,  there  are  still  some  areas  of 
the  program,  (i.e.,  destruction  of  secondary  hazardous  waste  and  closure  of  the  chemical 
weapons  destruction  facilities),  with  limited  information  to  adjust  cost  and  schedule  estimates. 

As  the  GAO  stated  in  their  final  report:  NSIAD-00-80,  Chemical  Weapons  Disposal: 
Improvements  Needed  in  Program  Accountability  and  Financial  Management,  May  2000:  “The 
Army  also  needs  additional  time  to  develop  a  reliable  baseline  to  estimate  the  costs  for  the 
closure  and  remediation  of  the  chemical  stockpile  disposal  facilities.  ...Furthermore,  because  no 
stockpile  disposal  facility  has  yet  to  be  remediated,  the  Army  lacks  real  time  experience  on 
which  to  estimate  these  costs.”  The  CDP  now  has  two  sites  to  compare  closure  activities, 
Johnston  Atoll  Chemical  Agent  Destruction  System  (JACADS)  and  Aberdeen  Chemical  Agent 
Disposal  Facility  (ABCDF).  JACADS  completed  destruction  operations  in  November  2000  and 
most  closure  activities  in  June  2004,  with  the  exception  of  final  closure  of  the  Resource 
Conservation  and  Recovery  Act  permit,  which  is  not  expected  until  November  2008.  The 
ABCDF  completed  operations  in  February  2006  and  closure  activities  were  completed  in  March 
2007.  With  this  limited  information  for  closure,  the  U.S.  Army  Chemical  Materials  Agency 
(CMA)  has  recently  completed  contract  performance  baselines  that  include  operations, 
secondary  waste  processing,  closure  and  known  risk,  for  sites  at  Anniston,  Alabama;  Pine  Bluff, 
Arkansas;  and  Umatilla,  Oregon. 

In  addition,  the  Department  approved  the  revised  Acquisition  Program  Baselines  for  the 
Chemical  Demilitarization  Program  -  U.S.  Army  Chemical  Materials  Agency  (Chem  Demil- 
CMA)  and  Chem  Demil-CMA  Newport  Major  Defense  Acquisition  Programs  in  April  2006, 
with  the  cost  and  schedule  estimates  based  on  the  Cost  Analysis  Improvement  Group’s  (CAIG) 
estimates.  The  CAIG  estimates  were  certified  to  Congress  in  May  2005  as  part  of  the  Nunn- 
McCurdy  unit  cost  breach.  The  increase  in  the  unit  cost  was  due  primarily  to  more  realistic 
schedule  estimates. 

Therefore,  the  Department  does  not  believe  it  is  prudent  to  adjust  cost  and  schedule  estimates  or 
rebaseline  at  this  time  due  to  limited  information  gained  during  the  past  year  associated  with 
processing  rates,  unknown  risks,  and  the  potential  for  a  chemical  incident.  The  Department  will 
evaluate  the  cost  and  schedule  estimates  for  the  entire  CDP  during  the  upcoming  Fiscal  Year 
2010-2015  Program  Objectives  Memorandum  build  process  and  program  review,  and  will 
determine  whether  an  update  is  warranted. 

RECOMMENDATION  5:  The  GAO  recommends  the  Secretary  of  Defense  direct  the  Chemical 
Analysis  Improvement  Group  to  establish  a  periodic  schedule  to  review  the  program’s 
processing  history  to  determine  whether  the  schedule  milestones  are  still  reasonable. 

DoD  RESPONSE:  Concur.  The  Department  conducts  a  biannual  program  review  of  the 
Chemical  Demilitarization  Program  in  support  of  the  budget  estimate  submission.  During  this 
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review,  the  program  office  schedule  and  cost  estimates  are  reviewed  in  comparison  with  the 
approved  Acquisition  Program  Baselines  to  determine  whether  there  are  any  significant 
differences.  If  there  are  significant  differences,  the  Department  will  request  the  Cost  Analysis 
Improvement  Group  conduct  an  independent  assessment  and  possibly  develop  cost  and  schedule 
estimates.  The  next  review  will  occur  during  the  Fiscal  Year  2010-2015  Program  Objectives 
Memorandum  build  process  and  program  review  in  the  spring  of  calendar  year  2008. 

RECOMMENDATION  6:  The  GAO  recommends  the  Secretary  of  Defense  direct  the  Under 
Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics  to  include  in  the  Chemical 
Demilitarization  Program’s  Acquisition  Program  Baseline  and  Selected  Acquisition  Report  the 
updated  and  reviewed  scheduled  milestones  for  completing  destruction  operations  and  facility 
closure  activities  for  each  of  the  chemical  agent  destruction  sites. 

DoD  RESPONSE:  Concur.  The  Department  will  request  administrative  changes  to  the 
Chemical  Demilitarization  Program-U.S.  Army  Chemical  Materials  Agency  (Chem  Demil  - 
CMA)  and  Chem  Demil-CMA  Newport  Acquisition  Program  Baselines  and  upcoming  Selected 
Acquisition  Reports  to  remove  the  “To  Be  Determined  (TBDs),”  which  were  used  as  a 
placeholder  for  schedule  milestones  beyond  April  2012,  and  replace  with  the  Cost  Analysis 
Improvement  Group’s  schedule  estimates. 

RECOMMENDATION  7:  The  GAO  recommends  the  Secretary  of  Defense  direct  the  Director 
of  the  U.S.  Army  Chemical  Materials  Agency  to  coordinate  with  the  systems  contractors  in 
defining  closure  requirements  and  developing  accurate  and  realistic  closure  cost  estimates  for 
each  of  DoD’s  chemical  demilitarization  sites. 

DoD  RESPONSE:  Non-Concur.  The  Department  recommends  GAO  delete  this 
recommendation  since  Recommendation  #9  is  more  encompassing. 

RECOMMENDATION  8:  The  GAO  recommends  the  Secretary  of  Defense  direct  the  Cost 
Analysis  Improvement  Group  to  conduct  an  independent  cost  review  of  the  cost  estimates. 

DoD  RESPONSE:  Partially  Concur.  The  Department  conducts  a  biannual  program  review  of 
the  Chemical  Demilitarization  Program  in  support  of  the  budget  estimate  submission.  During 
this  review,  the  program  office  schedule  and  cost  estimates  are  reviewed  in  comparison  with  the 
approved  Acquisition  Program  Baselines  to  determine  whether  there  are  any  significant 
differences.  If  there  are  significant  differences,  the  Department  will  request  the  Cost  Analysis 
Improvement  Group  conduct  an  independent  assessment  and  possibly  develop  cost  and  schedule 
estimates.  Therefore,  no  direction  to  conduct  a  cost  review  will  be  given  at  this  time. 

RECOMMENDATION  9:  The  GAO  recommends  the  Secretary  of  Defense  direct  the  Director 
of  the  U.S.  Army  Chemical  Materials  Agency  to  define  all  contract  requirements,  including  those 
associated  with  facility  closure  activities,  and  develop  comprehensive  cost  estimates  through 
closure  at  each  chemical  destruction  site. 

DoD  RESPONSE:  Partially  Concur.  Since  June  2007,  three  of  the  five  U.S.  Army  Chemical 
Materials  Agency  chemical  weapons  stockpile  destruction  sites  (Anniston,  Alabama;  Pine  Bluff, 
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Arkansas;  and  Umatilla,  Oregon)  have  completed  contract  performance  baselines  that  include 
contract  requirements  for  operations,  secondary  waste  processing,  closure  and  known  risk. 
Negotiations  are  under  way  to  complete  the  contract  performance  baseline  by  the  end  of  the  third 
quarter  of  Fiscal  Year  2008  for  the  Tooele,  Utah,  site.  Negotiations  are  expected  to  be 
completed  for  the  contract  requirements  and  cost  estimates  associated  with  the  closure  phase  for 
the  Newport,  Indiana,  site  by  the  end  of  the  third  quarter  of  Fiscal  Year  2008.  The  Department 
believes  no  further  direction  is  required. 

RECOMMENDATION  10:  The  GAO  recommends  the  Secretary  of  Defense  direct  the  Director 
of  the  U.S.  Army  Chemical  Materials  Agency  to  establish  a  time  frame  for  completing  the 
negotiation  for  placing  all  authorized  work  that  has  not  been  priced  on  contract  and  ensuring  that 
the  negotiations  take  place. 

DoD  RESPONSE:  Concur.  The  U.S.  Army  Chemical  Materials  Agency  has  negotiated  almost 
all  work  presently  on  contract  (current  and  future)  resulting  in  low  levels  of  authorized,  unpriced 
work.  The  Department  expects  negotiations  to  be  completed  by  the  end  of  the  third  quarter  of 
Fiscal  Year  2008. 

RECOMMENDATION  11:  The  GAO  recommends  the  Secretary  of  the  Defense  direct  the 
Director  of  the  U.S.  Army  Chemical  Materials  Agency  to  determine  for  each  of  the  four 
incineration  site  contracts  whether  greater  weight  can  be  applied  to  performance  measures,  such 
as  schedule  and  cost,  in  award  fee  plans  to  further  incentivize  the  systems  contractors  to  increase 
the  destmction  rate  of  chemical  agents  in  a  safe  and  environmentally  sound  manner. 

DoD  RESPONSE:  Concur.  The  Department  will  conduct  a  review  of  the  award  fee  plans  of  the 
U.S.  Army  Chemical  Materials  Agency  incineration  site  contracts  to  determine  whether  the 
weighting  factors  are  appropriate  to  incentivize  the  systems  contractors  to  reduce  the  schedule 
while  operating  in  a  safe,  secure,  and  environmentally-sound  manner.  The  results  will  be 
available  by  second  quarter  of  FY08. 

RECOMMENDATION  12:  The  GAO  recommends  the  Secretary  of  the  Defense  direct  the 
Director  of  the  U.S.  Army  Chemical  Materials  Agency  better  align  its  award  fee  policy  with 
DoD’s  March  2006  guidance  on  award  fees. 

DoD  RESPONSE:  Concur.  This  action  was  completed  on  September  7,  2007,  in  accordance 
with  the  March  29,  2006,  Memorandum:  Subject:  Award  Fee  Contracts,  from  the  Under 
Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics.  CMA  revised  Policy  Statement 
29,  “Procedure  for  Implementing  Contractor  Incentives  to  Include  Award  Fee  (AF)  and 
Performance  Based  Incentive  (PBI)  Fee.” 

RECOMMENDATION  13:  The  GAO  recommends  the  Secretary  of  the  Defense  direct  the 
Director  of  the  U.S.  Army  Chemical  Materials  Agency  to  link  more  of  its  award  fee  criteria  to 
performance  measures  that  focus  on  identifiable  interim  outcomes,  discrete  events,  or  milestones. 

DoD  RESPONSE:  Partially  Concur.  The  Department  will  conduct  a  review  of  the  U.S.  Army 
Chemical  Materials  Agency  (CMA)  award  fee  plans  to  determine  whether  there  are  performance 
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measures  that  focus  on  outcomes  or  outputs.  This  review  will  be  completed  along  with  the 
review  of  the  award  fee  weighting  factors  by  the  end  of  the  second  quarter  of  Fiscal  Year  2008. 
Based  on  the  findings  of  the  review,  the  Department  will  determine  whether  further  direction  and 
action  is  required.  Currently,  the  CMA  Fiscal  Year  2006  and  Fiscal  Year  2007  Award  Fee  plans 
included  objective  schedule  criteria  that  focused  on  U.S.  tons  of  chemical  agent  destroyed  to 
complete  destruction  by  April  2012. 
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